You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Nitay Joffe <ni...@actioniq.co> on 2015/09/10 15:16:03 UTC

DF.intersection issue in 1.5

The following fails for me in Spark 1.5:
https://gist.github.com/nitay/d08cb294ccf00b80c49a
Specifically, it returns 1 instead of 100 (in both versions).
When I print out the contents (i.e. collect()) I see all 100 items, yet the
count returns 1.

This works in 1.3 and 1.4.

Any ideas what's going on?

Re: DF.intersection issue in 1.5

Posted by Michael Armbrust <mi...@databricks.com>.

Thanks for pointing this out.
https://issues.apache.org/jira/browse/SPARK-10539

We will fix this for Spark 1.5.1.

On Thu, Sep 10, 2015 at 6:16 AM, Nitay Joffe <ni...@actioniq.co> wrote:

> The following fails for me in Spark 1.5:
> https://gist.github.com/nitay/d08cb294ccf00b80c49a
> Specifically, it returns 1 instead of 100 (in both versions).
> When I print out the contents (i.e. collect()) I see all 100 items, yet
> the count returns 1.
>
> This works in 1.3 and 1.4.
>
> Any ideas what's going on?
>