[jira] [Created] (SPARK-17756) java.lang.ClassCastException when using cartesian with DStream.transform

["Maciej Szymkiewicz (JIRA)" <ji...@apache.org>]

Maciej Szymkiewicz created SPARK-17756:
------------------------------------------

             Summary: java.lang.ClassCastException when using cartesian with DStream.transform
                 Key: SPARK-17756
                 URL: https://issues.apache.org/jira/browse/SPARK-17756
             Project: Spark
          Issue Type: Bug
          Components: PySpark, Streaming
    Affects Versions: 2.0.0
            Reporter: Maciej Szymkiewicz


Steps to reproduce:

{code}

from pyspark.streaming import StreamingContext

ssc = StreamingContext(spark.sparkContext, 10)
(ssc
    .queueStream([sc.range(10)])
    .transform(lambda rdd: rdd.cartesian(rdd))
    .pprint())

ssc.start()

## 16/10/01 21:34:30 ERROR JobScheduler: Error generating jobs for time 1475350470000 ms
## java.lang.ClassCastException: org.apache.spark.api.java.JavaPairRDD ## cannot be cast to org.apache.spark.api.java.JavaRDD
## 	at com.sun.proxy.$Proxy15.call(Unknown Source)
##    ....
{code}

A dummy fix is to put {{map(lamba x: x)}} which suggests it is a problem similar to https://issues.apache.org/jira/browse/SPARK-16589



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org