You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Dhimant <dh...@gmail.com> on 2014/09/05 13:58:36 UTC
error: type mismatch while Union
Hi,
I am getting type mismatch error while union operation.
Can someone suggest solution ?
/ case class MyNumber(no: Int, secondVal: String) extends Serializable
with Ordered[MyNumber] {
override def toString(): String = this.no.toString + " " +
this.secondVal
override def compare(that: MyNumber): Int = this.no compare that.no
override def compareTo(that: MyNumber): Int = this.no compare that.no
def Equals(that: MyNumber): Boolean = {
(this.no == that.no) && (that match {
case MyNumber(n1, n2) => n1 == no && n2 == secondVal
case _ => false
})
}
}
val numbers = sc.parallelize(1 to 20, 10)
val firstRdd = numbers.map(new MyNumber(_, "A"))
val secondRDD = numbers.map(new MyNumber(_, "B"))
val numberRdd = firstRdd .union(secondRDD )
<console>:24: error: type mismatch;
found : org.apache.spark.rdd.RDD[MyNumber]
required: org.apache.spark.rdd.RDD[MyNumber]
val numberRdd = onenumberRdd.union(anotherRDD)/
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/error-type-mismatch-while-Union-tp13547.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: error: type mismatch while Union
Posted by Dhimant <dh...@gmail.com>.
Thank you Aaron for pointing out problem. This only happens when I run this
code in spark-shell but not when i submit the job.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/error-type-mismatch-while-Union-tp13547p13677.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: error: type mismatch while Union
Posted by Aaron Davidson <il...@gmail.com>.
Are you doing this from the spark-shell? You're probably running into
https://issues.apache.org/jira/browse/SPARK-1199 which should be fixed in
1.1.
On Sat, Sep 6, 2014 at 3:03 AM, Dhimant <dh...@gmail.com> wrote:
> I am using Spark version 1.0.2
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/error-type-mismatch-while-Union-tp13547p13618.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: error: type mismatch while Union
Posted by Dhimant <dh...@gmail.com>.
I am using Spark version 1.0.2
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/error-type-mismatch-while-Union-tp13547p13618.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: error: type mismatch while Union
Posted by Yana Kadiyska <ya...@gmail.com>.
Which version are you using -- I can reproduce your issue w/ 0.9.2 but not
with 1.0.1...so my guess is that it's a bug and the fix hasn't been
backported... No idea on a workaround though..
On Fri, Sep 5, 2014 at 7:58 AM, Dhimant <dh...@gmail.com> wrote:
> Hi,
> I am getting type mismatch error while union operation.
> Can someone suggest solution ?
>
> / case class MyNumber(no: Int, secondVal: String) extends Serializable
> with Ordered[MyNumber] {
> override def toString(): String = this.no.toString + " " +
> this.secondVal
> override def compare(that: MyNumber): Int = this.no compare that.no
> override def compareTo(that: MyNumber): Int = this.no compare
> that.no
> def Equals(that: MyNumber): Boolean = {
> (this.no == that.no) && (that match {
> case MyNumber(n1, n2) => n1 == no && n2 == secondVal
> case _ => false
> })
> }
> }
> val numbers = sc.parallelize(1 to 20, 10)
> val firstRdd = numbers.map(new MyNumber(_, "A"))
> val secondRDD = numbers.map(new MyNumber(_, "B"))
> val numberRdd = firstRdd .union(secondRDD )
> <console>:24: error: type mismatch;
> found : org.apache.spark.rdd.RDD[MyNumber]
> required: org.apache.spark.rdd.RDD[MyNumber]
> val numberRdd = onenumberRdd.union(anotherRDD)/
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/error-type-mismatch-while-Union-tp13547.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>