You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Valery Khamenya <kh...@gmail.com> on 2018/06/18 21:00:47 UTC

Dataframe vs Dataset dilemma: either Row parsing or no filter push-down

Hi Spark gurus,

I was surprised to read here:
https://stackoverflow.com/questions/50129411/why-is-predicate-pushdown-not-used-in-typed-dataset-api-vs-untyped-dataframe-ap

that filters are not pushed down in typed Datasets and one should rather
stick to Dataframes.

But writing code for groupByKey + mapGroups is a headache with Dataframes
if compared to typed Dataset. The former mostly doesn't force you to write
any Encoders (unless you try to write generic transformations on
parametrized Dataset[T]) . Neither typed Dataset forces you to do an ugly
Row parsing with getInteger, getString, etc -- like it is needed with
Dataframes.

So, what should the poor Spark user rely on by default, if the goal is to
deliver a library of  data transformations -- Dataset or Dataframe?

best regards
--
Valery

Re: Dataframe vs Dataset dilemma: either Row parsing or no filter push-down

Posted by Koert Kuipers <ko...@tresata.com>.
we use DataFrame and RDD. Dataset not only has issues with predicate
pushdown, it also adds shufffles at times where it shouldn't. and there is
some overhead from the encoders themselves, because under the hood it is
still just Row objects.


On Mon, Jun 18, 2018 at 5:00 PM, Valery Khamenya <kh...@gmail.com> wrote:

> Hi Spark gurus,
>
> I was surprised to read here:
> https://stackoverflow.com/questions/50129411/why-is-
> predicate-pushdown-not-used-in-typed-dataset-api-vs-untyped-dataframe-ap
>
> that filters are not pushed down in typed Datasets and one should rather
> stick to Dataframes.
>
> But writing code for groupByKey + mapGroups is a headache with Dataframes
> if compared to typed Dataset. The former mostly doesn't force you to write
> any Encoders (unless you try to write generic transformations on
> parametrized Dataset[T]) . Neither typed Dataset forces you to do an ugly
> Row parsing with getInteger, getString, etc -- like it is needed with
> Dataframes.
>
> So, what should the poor Spark user rely on by default, if the goal is to
> deliver a library of  data transformations -- Dataset or Dataframe?
>
> best regards
> --
> Valery
>