[jira] [Commented] (SPARK-1836) REPL $outer type mismatch causes lookup() and equals() problems

["Michael Armbrust (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998312#comment-13998312 ] 

Michael Armbrust commented on SPARK-1836:
-----------------------------------------

This sounds like it could be related to [SPARK-1199]

> REPL $outer type mismatch causes lookup() and equals() problems
> ---------------------------------------------------------------
>
>                 Key: SPARK-1836
>                 URL: https://issues.apache.org/jira/browse/SPARK-1836
>             Project: Spark
>          Issue Type: Bug
>    Affects Versions: 0.9.0
>            Reporter: Michael Malak
>
> Anand Avati partially traced the cause to REPL wrapping classes in $outer classes. There are at least two major symptoms:
> 1. equals()
> =========
> In REPL equals() (required in custom classes used as a key for groupByKey) seems to have to be written using instanceOf[] instead of the canonical match{}
> Spark Shell (equals uses match{}):
> {noformat}
> class C(val s:String) extends Serializable {
>   override def equals(o: Any) = o match {
>     case that: C => that.s == s
>     case _ => false
>   }
> }
> val x = new C("a")
> val bos = new java.io.ByteArrayOutputStream()
> val out = new java.io.ObjectOutputStream(bos)
> out.writeObject(x);
> val b = bos.toByteArray();
> out.close
> bos.close
> val y = new java.io.ObjectInputStream(new ava.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
> x.equals(y)
> res: Boolean = false
> {noformat}
> Spark Shell (equals uses isInstanceOf[]):
> {noformat}
> class C(val s:String) extends Serializable {
>   override def equals(o: Any) = if (o.isInstanceOf[C]) (o.asInstanceOf[C].s = s) else false
> }
> val x = new C("a")
> val bos = new java.io.ByteArrayOutputStream()
> val out = new java.io.ObjectOutputStream(bos)
> out.writeObject(x);
> val b = bos.toByteArray();
> out.close
> bos.close
> val y = new java.io.ObjectInputStream(new ava.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
> x.equals(y)
> res: Boolean = true
> {noformat}
> Scala Shell (equals uses match{}):
> {noformat}
> class C(val s:String) extends Serializable {
>   override def equals(o: Any) = o match {
>     case that: C => that.s == s
>     case _ => false
>   }
> }
> val x = new C("a")
> val bos = new java.io.ByteArrayOutputStream()
> val out = new java.io.ObjectOutputStream(bos)
> out.writeObject(x);
> val b = bos.toByteArray();
> out.close
> bos.close
> val y = new java.io.ObjectInputStream(new java.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
> x.equals(y)
> res: Boolean = true
> {noformat}
> 2. lookup()
> =========
> {noformat}
> class C(val s:String) extends Serializable {
>   override def equals(o: Any) = if (o.isInstanceOf[C]) o.asInstanceOf[C].s == s else false
>   override def hashCode = s.hashCode
>   override def toString = s
> }
> val r = sc.parallelize(Array((new C("a"),11),(new C("a"),12)))
> r.lookup(new C("a"))
> <console>:17: error: type mismatch;
>  found   : C
>  required: C
>               r.lookup(new C("a"))
>                        ^
> {noformat}
> See
> http://mail-archives.apache.org/mod_mbox/spark-dev/201405.mbox/%3C1400019424.80629.YahooMailNeo%40web160801.mail.bf1.yahoo.com%3E



--
This message was sent by Atlassian JIRA
(v6.2#6252)