You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Shiyuan <gs...@gmail.com> on 2017/04/08 18:15:18 UTC

Why dataframe can be more efficient than dataset?

Hi Spark-users,
    I came across a few sources which mentioned DataFrame can be more
efficient than Dataset.  I can understand this is true because Dataset
allows functional transformation which Catalyst cannot look into and hence
cannot optimize well. But can DataFrame be more efficient than Dataset even
if we only use the relational transformation on dataset? If so, can anyone
give some explanation why  it is so? Any benchmark comparing dataset vs.
dataframe?   Thank you!

Shiyuan

Re: Why dataframe can be more efficient than dataset?

Posted by Koert Kuipers <ko...@tresata.com>.
let me try that again. i left some crap at the bottom of my previous email
as i was editing it. sorry about that. here it goes:

it is because you use Dataset[X] but the actual computations are still done
in Dataset[Row] (so DataFrame). well... the actual computations are done in
RDD[InternalRow] with spark's internal types to represent String, Map, Seq,
structs, etc.

so for example if you do:
scala> val x: Dataset[(String, String)] = ...
scala> val f: (String, String) => Boolean = _._2 != null
scala> x.filter(f)

in this case you are using a lambda function for the filter. this is a
black-box operation to spark (spark cannot see what is inside the
function). so spark will now convert the internal representation it is
actually using (something like an InternalRow of size 2 with inside of it
two objects of type UTF8String) into a Tuple2[String, String], and then
call your function f on it. so for this very simply null comparison you are
doing a relatively expensive conversion.

now compare this to if you have a DataFrame that holds 2 columns of type
String.
scala> val x: DataFrame = ...
x: org.apache.spark.sql.DataFrame = [x: string, y: string]
scala> x.filter($"y" isNotNull)

spark will parse your expression, and since it has an understanding of what
you are trying to do, it can apply the logic directly on the InternalRow,
which avoids the conversion. this will be faster. of course you pay the
price for this in that you are forced to use a much more constrained
framework to express what you want to do, which can lead to some hair
pulling at times.

On Sat, Apr 8, 2017 at 2:15 PM, Shiyuan <gs...@gmail.com> wrote:

> Hi Spark-users,
>     I came across a few sources which mentioned DataFrame can be more
> efficient than Dataset.  I can understand this is true because Dataset
> allows functional transformation which Catalyst cannot look into and hence
> cannot optimize well. But can DataFrame be more efficient than Dataset even
> if we only use the relational transformation on dataset? If so, can anyone
> give some explanation why  it is so? Any benchmark comparing dataset vs.
> dataframe?   Thank you!
>
> Shiyuan
>

Re: Why dataframe can be more efficient than dataset?

Posted by Koert Kuipers <ko...@tresata.com>.
it is because you use Dataset[X] but the actual computations are still done
in Dataset[Row] (so DataFrame). well... the actual computations are done in
RDD[InternalRow] with spark's internal types to represent String, Map, Seq,
structs, etc.

so for example if you do:
scala> val x: Dataset[(String, String)] = ...
scala> val f: (String, String) => Boolean = _._2 != null
scala> x.filter(f)

in this case you are using a lambda function for the filter. this is a
black-box operation to spark (spark cannot see what is inside the
function). so spark will now convert the internal representation it is
actually using (something like an InternalRow of size 2 with inside of it
two objects of type UTF8String) into a Tuple2[String, String], and then
call your function f on it. so for this very simply null comparison you are
doing a relatively expensive conversion.

now compare this to if you have a DataFrame that holds 2 columns of type
String.
scala> val x: DataFrame = ...
x: org.apache.spark.sql.DataFrame = [x: string, y: string]
scala> x.filter($"y" isNotNull)

spark will parse your expression, and since it has an understanding of what
you are trying to do, it can apply the logic directly on the InternalRow,
which avoids the conversion. this will be faster. of course you pay the
price for this in that you are forced to use a much more constrained
framework to express what you want to do, which can lead to some hair
pulling at times.



so when you do a lambda operation on type X, this is black

want to use X spark needs to convert these InternalRows to X and then
convert the result back to InternalRows.



 (so DataFrame) using spark's internal types for string, seq, map, etc.
so any time you actually need an X there is conversion from Row to X, and
from internal representations to your representations of the data) and back
going on. this is whats the encoders are used for.

2) some optimizations aren't working yet for Dataset[X]
3) since type X and the lambdas that you define that perform on it are
somewhat of a black box to spark there is less room for optimization.



On Sat, Apr 8, 2017 at 2:15 PM, Shiyuan <gs...@gmail.com> wrote:

> Hi Spark-users,
>     I came across a few sources which mentioned DataFrame can be more
> efficient than Dataset.  I can understand this is true because Dataset
> allows functional transformation which Catalyst cannot look into and hence
> cannot optimize well. But can DataFrame be more efficient than Dataset even
> if we only use the relational transformation on dataset? If so, can anyone
> give some explanation why  it is so? Any benchmark comparing dataset vs.
> dataframe?   Thank you!
>
> Shiyuan
>

Re: Why dataframe can be more efficient than dataset?

Posted by DB Tsai <db...@dbtsai.com>.
There is a JIRA and prototype which analyzes the JVM bytecode in the black
box, and convert the closures into catalyst expressions.

https://issues.apache.org/jira/browse/SPARK-14083

This potentially can address the issue discussed here.


Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 0x5CED8B896A6BDFA0

On Sun, Apr 9, 2017 at 11:17 AM, Koert Kuipers <ko...@tresata.com> wrote:

> in this case there is no difference in performance. both will do the
> operation directly on the internal representation of the data (so the
> InternalRow).
>
> also it is worth pointing out that switching back and forth between
> Dataset[X] and DataFrame is free.
>
> On Sun, Apr 9, 2017 at 1:28 PM, Shiyuan <gs...@gmail.com> wrote:
>
>> Thank you for the detailed explanation!  You point out two reasons why
>> Dataset is not as efficeint as dataframe:
>> 1). Spark cannot look into lambda and therefore cannot optimize.
>> 2). The  type conversion  occurs under the hood, eg. from X to internal
>> row.
>>
>> Just to check my understanding,  some method of Dataset can also take sql
>> expression string  instead of lambda function, in this case, Is it  the
>> type conversion still happens under the hood and therefore Dataset is still
>> not as efficient as DataFrame.  Here is the code,
>>
>> //define a dataset and a dataframe, same content, but one is stored as
>> Dataset<Person>, the other is Dataset<Row>
>> scala> case class Person(name: String, age: Long)
>> scala> val ds = Seq(Person("A",32), Person("B", 18)).toDS
>> ds: org.apache.spark.sql.Dataset[Person] = [name: string, age: bigint]
>> scala> val df = Seq(Person("A",32), Person("B", 18)).toDF
>> df: org.apache.spark.sql.DataFrame = [name: string, age: bigint]
>>
>> //Which filtering is more efficient? both use sql expression string.
>> scala> df.filter("age < 20")
>> res7: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [name:
>> string, age: bigint]
>>
>> scala> ds.filter("age < 20")
>> res8: org.apache.spark.sql.Dataset[Person] = [name: string, age: bigint]
>>
>>
>>
>>
>>
>>
>>
>>
>> On Sat, Apr 8, 2017 at 7:22 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>
>>> how would you use only relational transformations on dataset?
>>>
>>> On Sat, Apr 8, 2017 at 2:15 PM, Shiyuan <gs...@gmail.com> wrote:
>>>
>>>> Hi Spark-users,
>>>>     I came across a few sources which mentioned DataFrame can be more
>>>> efficient than Dataset.  I can understand this is true because Dataset
>>>> allows functional transformation which Catalyst cannot look into and hence
>>>> cannot optimize well. But can DataFrame be more efficient than Dataset even
>>>> if we only use the relational transformation on dataset? If so, can anyone
>>>> give some explanation why  it is so? Any benchmark comparing dataset vs.
>>>> dataframe?   Thank you!
>>>>
>>>> Shiyuan
>>>>
>>>
>>>
>>
>

Re: Why dataframe can be more efficient than dataset?

Posted by Koert Kuipers <ko...@tresata.com>.
in this case there is no difference in performance. both will do the
operation directly on the internal representation of the data (so the
InternalRow).

also it is worth pointing out that switching back and forth between
Dataset[X] and DataFrame is free.

On Sun, Apr 9, 2017 at 1:28 PM, Shiyuan <gs...@gmail.com> wrote:

> Thank you for the detailed explanation!  You point out two reasons why
> Dataset is not as efficeint as dataframe:
> 1). Spark cannot look into lambda and therefore cannot optimize.
> 2). The  type conversion  occurs under the hood, eg. from X to internal
> row.
>
> Just to check my understanding,  some method of Dataset can also take sql
> expression string  instead of lambda function, in this case, Is it  the
> type conversion still happens under the hood and therefore Dataset is still
> not as efficient as DataFrame.  Here is the code,
>
> //define a dataset and a dataframe, same content, but one is stored as
> Dataset<Person>, the other is Dataset<Row>
> scala> case class Person(name: String, age: Long)
> scala> val ds = Seq(Person("A",32), Person("B", 18)).toDS
> ds: org.apache.spark.sql.Dataset[Person] = [name: string, age: bigint]
> scala> val df = Seq(Person("A",32), Person("B", 18)).toDF
> df: org.apache.spark.sql.DataFrame = [name: string, age: bigint]
>
> //Which filtering is more efficient? both use sql expression string.
> scala> df.filter("age < 20")
> res7: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [name:
> string, age: bigint]
>
> scala> ds.filter("age < 20")
> res8: org.apache.spark.sql.Dataset[Person] = [name: string, age: bigint]
>
>
>
>
>
>
>
>
> On Sat, Apr 8, 2017 at 7:22 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
>> how would you use only relational transformations on dataset?
>>
>> On Sat, Apr 8, 2017 at 2:15 PM, Shiyuan <gs...@gmail.com> wrote:
>>
>>> Hi Spark-users,
>>>     I came across a few sources which mentioned DataFrame can be more
>>> efficient than Dataset.  I can understand this is true because Dataset
>>> allows functional transformation which Catalyst cannot look into and hence
>>> cannot optimize well. But can DataFrame be more efficient than Dataset even
>>> if we only use the relational transformation on dataset? If so, can anyone
>>> give some explanation why  it is so? Any benchmark comparing dataset vs.
>>> dataframe?   Thank you!
>>>
>>> Shiyuan
>>>
>>
>>
>

Re: Why dataframe can be more efficient than dataset?

Posted by Shiyuan <gs...@gmail.com>.
Thank you for the detailed explanation!  You point out two reasons why
Dataset is not as efficeint as dataframe:
1). Spark cannot look into lambda and therefore cannot optimize.
2). The  type conversion  occurs under the hood, eg. from X to internal
row.

Just to check my understanding,  some method of Dataset can also take sql
expression string  instead of lambda function, in this case, Is it  the
type conversion still happens under the hood and therefore Dataset is still
not as efficient as DataFrame.  Here is the code,

//define a dataset and a dataframe, same content, but one is stored as
Dataset<Person>, the other is Dataset<Row>
scala> case class Person(name: String, age: Long)
scala> val ds = Seq(Person("A",32), Person("B", 18)).toDS
ds: org.apache.spark.sql.Dataset[Person] = [name: string, age: bigint]
scala> val df = Seq(Person("A",32), Person("B", 18)).toDF
df: org.apache.spark.sql.DataFrame = [name: string, age: bigint]

//Which filtering is more efficient? both use sql expression string.
scala> df.filter("age < 20")
res7: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [name:
string, age: bigint]

scala> ds.filter("age < 20")
res8: org.apache.spark.sql.Dataset[Person] = [name: string, age: bigint]








On Sat, Apr 8, 2017 at 7:22 PM, Koert Kuipers <ko...@tresata.com> wrote:

> how would you use only relational transformations on dataset?
>
> On Sat, Apr 8, 2017 at 2:15 PM, Shiyuan <gs...@gmail.com> wrote:
>
>> Hi Spark-users,
>>     I came across a few sources which mentioned DataFrame can be more
>> efficient than Dataset.  I can understand this is true because Dataset
>> allows functional transformation which Catalyst cannot look into and hence
>> cannot optimize well. But can DataFrame be more efficient than Dataset even
>> if we only use the relational transformation on dataset? If so, can anyone
>> give some explanation why  it is so? Any benchmark comparing dataset vs.
>> dataframe?   Thank you!
>>
>> Shiyuan
>>
>
>

Re: Why dataframe can be more efficient than dataset?

Posted by Koert Kuipers <ko...@tresata.com>.
how would you use only relational transformations on dataset?

On Sat, Apr 8, 2017 at 2:15 PM, Shiyuan <gs...@gmail.com> wrote:

> Hi Spark-users,
>     I came across a few sources which mentioned DataFrame can be more
> efficient than Dataset.  I can understand this is true because Dataset
> allows functional transformation which Catalyst cannot look into and hence
> cannot optimize well. But can DataFrame be more efficient than Dataset even
> if we only use the relational transformation on dataset? If so, can anyone
> give some explanation why  it is so? Any benchmark comparing dataset vs.
> dataframe?   Thank you!
>
> Shiyuan
>

Re: Why dataframe can be more efficient than dataset?

Posted by Jörn Franke <jo...@gmail.com>.
As far as I am aware in newer Spark versions a DataFrame is the same as Dataset[Row].
In fact, performance depends on so many factors, so I am not sure such a comparison makes sense.

> On 8. Apr 2017, at 20:15, Shiyuan <gs...@gmail.com> wrote:
> 
> Hi Spark-users, 
>     I came across a few sources which mentioned DataFrame can be more efficient than Dataset.  I can understand this is true because Dataset allows functional transformation which Catalyst cannot look into and hence cannot optimize well. But can DataFrame be more efficient than Dataset even if we only use the relational transformation on dataset? If so, can anyone give some explanation why  it is so? Any benchmark comparing dataset vs. dataframe?   Thank you!
> 
> Shiyuan 

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org