You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Axel Dahl <ax...@whisperstream.com> on 2015/09/08 22:22:26 UTC

performance when checking if data frame is empty or not

I have a join, that fails when one of the data frames is empty.

To avoid this I am hoping to check if the dataframe is empty or not before
the join.

The question is what's the most performant way to do that?

should I do df.count() or df.first() or something else?

Thanks in advance,

-Axel

Re: performance when checking if data frame is empty or not

Posted by Ted Yu <yu...@gmail.com>.

Have you tried:

df.rdd.isEmpty

Cheers

On Tue, Sep 8, 2015 at 1:22 PM, Axel Dahl <ax...@whisperstream.com> wrote:

> I have a join, that fails when one of the data frames is empty.
>
> To avoid this I am hoping to check if the dataframe is empty or not before
> the join.
>
> The question is what's the most performant way to do that?
>
> should I do df.count() or df.first() or something else?
>
> Thanks in advance,
>
> -Axel
>