[jira] [Created] (SPARK-16589) Chained cartesian produces incorrect number of records

["Maciej Szymkiewicz (JIRA)" <ji...@apache.org>]

Maciej Szymkiewicz created SPARK-16589:
------------------------------------------

             Summary: Chained cartesian produces incorrect number of records
                 Key: SPARK-16589
                 URL: https://issues.apache.org/jira/browse/SPARK-16589
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 1.6.0, 2.0.0
            Reporter: Maciej Szymkiewicz


Chaining cartesian calls in PySpark results in the number of records lower than expected. It can be reproduced as follows:

{code}
rdd = sc.parallelize(range(10), 1)
rdd.cartesian(rdd).cartesian(rdd).count()
## 355
{code}

It looks like it is related to serialization. If we reserialize after initial cartesian:

{code}
rdd.cartesian(rdd)._reserialize(BatchedSerializer(PickleSerializer(), 1)).cartesian(rdd).count()
{code}

or insert identity map:

{code}
rdd.cartesian(rdd).map(lambda x: x).cartesian(rdd).count()
{code}

it yields correct results.
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org