You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2016/02/27 10:40:30 UTC
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/11410
[SPARK-13456][SQL] fix creating encoders for case classes defined in Spark shell
## What changes were proposed in this pull request?
case classes defined in REPL are wrapped by line classes, and we have a trick for scala 2.10 REPL to automatically register the wrapper classes to `OuterScope` so that we can use when create encoders.
However, this trick doesn't work right after we upgrade to scala 2.11, and unfortunately the tests are only in scala 2.10, which makes this bug hidden until now.
This PR moves the encoder tests to scala 2.11 `ReplSuite`, and fixes this bug by another approach(the previous trick can't port to scala 2.11 REPL): make `OuterScope` smarter that can detect classes defined in REPL and load line wrapper classes automatically.
## How was this patch tested?
the migrated encoder tests in `ReplSuite`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark repl
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11410.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11410
----
commit a2c461746b10fab4acac5e206a759a9bebbc7c29
Author: Wenchen Fan <we...@databricks.com>
Date: 2016-02-26T13:09:34Z
fix creating encoders for case classes defined in Spark shell
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-191079474
**[Test build #52286 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52286/consoleFull)** for PR 11410 at commit [`a0510c4`](https://github.com/apache/spark/commit/a0510c42b6a63468d16f80b5cc81b7367d8c67a2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` // `INSTANCE()` method to get the single instance of class `$read`. Then call `$iw()``
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189784923
**[Test build #52137 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52137/consoleFull)** for PR 11410 at commit [`a9d832b`](https://github.com/apache/spark/commit/a9d832bb9e2f92bf5fb35a7d7283785f4c952304).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-196642468
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by ScrapCodes <gi...@git.apache.org>.
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/11410#discussion_r54401926
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/OuterScopes.scala ---
@@ -39,4 +41,35 @@ object OuterScopes {
def addOuterScope(outer: AnyRef): Unit = {
outerScopes.putIfAbsent(outer.getClass.getName, outer)
}
+
+ def getOuterScope(innerCls: Class[_]): AnyRef = {
+ assert(innerCls.isMemberClass)
+ val outerCls = innerCls.getDeclaringClass.getName
+ val outer = outerScopes.get(outerCls)
+ if (outer == null) {
+ outerCls match {
+ case REPLClass(line) =>
+ val loader = Utils.getContextOrSparkClassLoader
+ def loadCls(clsName: String): Class[_] = Class.forName(clsName, true, loader)
+
+ val cls1 = loadCls(line + "$read$")
+ val obj1 = cls1.getField("MODULE$").get(null)
+
+ val obj2 = cls1.getMethod("INSTANCE").invoke(obj1)
+ val cls2 = loadCls(line + "$read")
+
+ val obj3 = cls2.getMethod("$iw").invoke(obj2)
+ val cls3 = loadCls(line + "$read$$iw")
+
+ cls3.getMethod("$iw").invoke(obj3)
+
+ case _ => null
+ }
+ } else {
+ outer
+ }
+ }
+
+ // The format of REPL generated wrapper class's name, e.g. `$line12.$read$$iw$$iw`
--- End diff --
AFAIK this is going to be okay, and very less likely to change I suppose. I have not tested your change.
This is an interesting patch, I will spend more time looking at it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189667873
**[Test build #52129 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52129/consoleFull)** for PR 11410 at commit [`a9d832b`](https://github.com/apache/spark/commit/a9d832bb9e2f92bf5fb35a7d7283785f4c952304).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199156935
**[Test build #53658 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53658/consoleFull)** for PR 11410 at commit [`47e0e39`](https://github.com/apache/spark/commit/47e0e393c907ffa54ac23c9902f5db00dcbca147).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199238467
cc @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-196675771
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53157/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189660066
**[Test build #52126 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52126/consoleFull)** for PR 11410 at commit [`a0a8424`](https://github.com/apache/spark/commit/a0a8424b33ff147023504ead0168ce05b64310f4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-196690093
Hi @retronym , your idea looks good, but in our situation, we only have a class name string and it's not that easy to get the line object. BTW the REPL classes are loaded dynamically in Spark, I think we have to use reflection to work on them. Any ideas?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-191153397
**[Test build #52303 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52303/consoleFull)** for PR 11410 at commit [`a0510c4`](https://github.com/apache/spark/commit/a0510c42b6a63468d16f80b5cc81b7367d8c67a2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` // `INSTANCE()` method to get the single instance of class `$read`. Then call `$iw()``
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-193662397
**[Test build #52650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52650/consoleFull)** for PR 11410 at commit [`8b79215`](https://github.com/apache/spark/commit/8b7921582868a7b9298385bcc11f6f0547969de0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189613891
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52115/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189686559
**[Test build #52129 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52129/consoleFull)** for PR 11410 at commit [`a9d832b`](https://github.com/apache/spark/commit/a9d832bb9e2f92bf5fb35a7d7283785f4c952304).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` // `INSTANCE()` method to get the single instance of class `$read`. Then call `$iw()``
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-191098934
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189686635
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by retronym <gi...@git.apache.org>.
Github user retronym commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-196676313
Since offering review, I've found a more direct way to get to the object.
```
Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_71).
Type in expressions for evaluation. Or try :help.
scala> $intp.isettings.unwrapStrings = false
$intp.isettings.unwrapStrings: Boolean = false
scala> var foo = ""
foo: String = ""
scala> foo = "2"
foo: String = 2
scala> val last = $intp.lastRequest
last: $intp.Request = Request(line=foo = "2", 1 trees)
scala> last.fullAccessPath
res1: String = $line4.$read.$iw.$iw
scala> val iw = $line4.$read.$iw.$iw
iw: $line4.$read.$iw.$iw.type = $line4.$read$$iw$$iw$@23444a36
scala> iw.getClass.getMethods.find(_.getName.startsWith("$ires")).head.invoke(iw)
res2: Object = 2
scala> def nullIRes(x: Any) = { val cls = x.getClass; cls.getDeclaredFields.filter(_.getName.startsWith("$ires")).foreach{fld => fld.setAccessible(true); fld.set(x, null)}}
nullIRes: (x: Any)Unit
scala> $intp.interpret(s"nullIRes(${last.fullAccessPath})")
res4: scala.tools.nsc.interpreter.IR.Result = Success
scala> iw.getClass.getMethods.find(_.getName.startsWith("$ires")).head.invoke(iw)
res6: Object = null
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-194071403
**[Test build #52715 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52715/consoleFull)** for PR 11410 at commit [`9be89ff`](https://github.com/apache/spark/commit/9be89ffaa91f8eaef62397db3314d89e0b04fc18).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-196675770
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-193982927
@retronym Will have time to take another round of review? Thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-198457438
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53541/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189656049
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52119/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by retronym <gi...@git.apache.org>.
Github user retronym commented on a diff in the pull request:
https://github.com/apache/spark/pull/11410#discussion_r54647099
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/OuterScopes.scala ---
@@ -39,4 +41,35 @@ object OuterScopes {
def addOuterScope(outer: AnyRef): Unit = {
outerScopes.putIfAbsent(outer.getClass.getName, outer)
}
+
+ def getOuterScope(innerCls: Class[_]): AnyRef = {
+ assert(innerCls.isMemberClass)
+ val outerCls = innerCls.getDeclaringClass.getName
+ val outer = outerScopes.get(outerCls)
+ if (outer == null) {
+ outerCls match {
+ case REPLClass(line) =>
+ val loader = Utils.getContextOrSparkClassLoader
+ def loadCls(clsName: String): Class[_] = Class.forName(clsName, true, loader)
+
+ val cls1 = loadCls(line + "$read$")
+ val obj1 = cls1.getField("MODULE$").get(null)
+
+ val obj2 = cls1.getMethod("INSTANCE").invoke(obj1)
+ val cls2 = loadCls(line + "$read")
+
+ val obj3 = cls2.getMethod("$iw").invoke(obj2)
+ val cls3 = loadCls(line + "$read$$iw")
+
+ cls3.getMethod("$iw").invoke(obj3)
+
+ case _ => null
+ }
+ } else {
+ outer
+ }
+ }
+
+ // The format of REPL generated wrapper class's name, e.g. `$line12.$read$$iw$$iw`
--- End diff --
Yes, that's exactly right. Look at the generate code in the `$eval` object (again, by running with `-Xprint:parser`)
```scala
package $line10 {
object $eval extends scala.AnyRef {
def <init>() = {
super.<init>();
()
};
lazy val $result = $line10.$read.INSTANCE.$iw.$iw.$iw.$iw.res2;
lazy val $print: String = {
$line10.$read.INSTANCE.$iw.$iw.$iw.$iw;
"".$plus("i: Int = ").$plus(scala.runtime.ScalaRunTime.replStringOf($line10.$read.INSTANCE.$iw.$iw.$iw.$iw.i, 1000)).$plus("res2: String = ").$plus(scala.runtime.ScalaRunTime.replStringOf($line10.$read.INSTANCE.$iw.$iw.$iw.$iw.res2, 1000))
}
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189621755
**[Test build #52116 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52116/consoleFull)** for PR 11410 at commit [`a0a8424`](https://github.com/apache/spark/commit/a0a8424b33ff147023504ead0168ce05b64310f4).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189686637
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52129/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199211759
**[Test build #53666 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53666/consoleFull)** for PR 11410 at commit [`47e0e39`](https://github.com/apache/spark/commit/47e0e393c907ffa54ac23c9902f5db00dcbca147).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11410#discussion_r54331261
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/OuterScopes.scala ---
@@ -39,4 +41,35 @@ object OuterScopes {
def addOuterScope(outer: AnyRef): Unit = {
outerScopes.putIfAbsent(outer.getClass.getName, outer)
}
+
+ def getOuterScope(innerCls: Class[_]): AnyRef = {
+ assert(innerCls.isMemberClass)
+ val outerCls = innerCls.getDeclaringClass.getName
+ val outer = outerScopes.get(outerCls)
+ if (outer == null) {
+ outerCls match {
+ case REPLClass(line) =>
+ val loader = Utils.getContextOrSparkClassLoader
+ def loadCls(clsName: String): Class[_] = Class.forName(clsName, true, loader)
+
+ val cls1 = loadCls(line + "$read$")
+ val obj1 = cls1.getField("MODULE$").get(null)
+
+ val obj2 = cls1.getMethod("INSTANCE").invoke(obj1)
+ val cls2 = loadCls(line + "$read")
+
+ val obj3 = cls2.getMethod("$iw").invoke(obj2)
+ val cls3 = loadCls(line + "$read$$iw")
+
+ cls3.getMethod("$iw").invoke(obj3)
+
+ case _ => null
+ }
+ } else {
+ outer
+ }
+ }
+
+ // The format of REPL generated wrapper class's name, e.g. `$line12.$read$$iw$$iw`
--- End diff --
I'm not sure if this is always true, need some experts to confirm.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11410#discussion_r54570439
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/OuterScopes.scala ---
@@ -39,4 +41,35 @@ object OuterScopes {
def addOuterScope(outer: AnyRef): Unit = {
outerScopes.putIfAbsent(outer.getClass.getName, outer)
}
+
+ def getOuterScope(innerCls: Class[_]): AnyRef = {
+ assert(innerCls.isMemberClass)
+ val outerCls = innerCls.getDeclaringClass.getName
+ val outer = outerScopes.get(outerCls)
+ if (outer == null) {
+ outerCls match {
+ case REPLClass(line) =>
+ val loader = Utils.getContextOrSparkClassLoader
+ def loadCls(clsName: String): Class[_] = Class.forName(clsName, true, loader)
+
+ val cls1 = loadCls(line + "$read$")
+ val obj1 = cls1.getField("MODULE$").get(null)
+
+ val obj2 = cls1.getMethod("INSTANCE").invoke(obj1)
+ val cls2 = loadCls(line + "$read")
+
+ val obj3 = cls2.getMethod("$iw").invoke(obj2)
+ val cls3 = loadCls(line + "$read$$iw")
+
+ cls3.getMethod("$iw").invoke(obj3)
+
+ case _ => null
+ }
+ } else {
+ outer
+ }
+ }
+
+ // The format of REPL generated wrapper class's name, e.g. `$line12.$read$$iw$$iw`
--- End diff --
If there are more than 2 nested `$iw` classes, can I call `$iw` method multiple times to get the singleton of the inner-most `$iw` class? e.g. `obj` is the singleton of `$line3.$read` class, can I use `obj.$iw().$iw().$iw().$iw()` to get the singleton of inner most `$iw` class?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199136200
**[Test build #53658 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53658/consoleFull)** for PR 11410 at commit [`47e0e39`](https://github.com/apache/spark/commit/47e0e393c907ffa54ac23c9902f5db00dcbca147).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199212229
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53666/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189656038
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-193744957
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-191153687
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52303/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11410#discussion_r54334648
--- Diff: repl/scala-2.10/src/test/scala/org/apache/spark/repl/ReplSuite.scala ---
@@ -288,7 +288,7 @@ class ReplSuite extends SparkFunSuite {
|import org.apache.spark.sql.Encoder
|import org.apache.spark.sql.expressions.Aggregator
|import org.apache.spark.sql.TypedColumn
- |val simpleSum = new Aggregator[Int, Int, Int] with Serializable {
+ |val simpleSum = new Aggregator[Int, Int, Int] {
--- End diff --
`Aggregator` already extends `Serializable`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-198457241
**[Test build #53541 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53541/consoleFull)** for PR 11410 at commit [`41c2cc5`](https://github.com/apache/spark/commit/41c2cc595e30bdc1237acd3e309050910758ca39).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189621792
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52116/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-190653205
**[Test build #52228 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52228/consoleFull)** for PR 11410 at commit [`f44e479`](https://github.com/apache/spark/commit/f44e479f57b794f635b89409c636afab85b70240).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189784873
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-191153683
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-190653742
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-194108195
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52715/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199212226
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-190603697
**[Test build #52228 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52228/consoleFull)** for PR 11410 at commit [`f44e479`](https://github.com/apache/spark/commit/f44e479f57b794f635b89409c636afab85b70240).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11410#discussion_r55296075
--- Diff: repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala ---
@@ -254,6 +254,30 @@ class ReplSuite extends SparkFunSuite {
|import sqlContext.implicits._
|case class TestCaseClass(value: Int)
|sc.parallelize(1 to 10).map(x => TestCaseClass(x)).toDF().collect()
+ |
+ |// Test Dataset Serialization in the REPL
+ |Seq(TestCaseClass(1)).toDS().collect()
+ """.stripMargin)
+ assertDoesNotContain("error:", output)
+ assertDoesNotContain("Exception", output)
+ }
+
+ test("Datasets and encoders") {
+ val output = runInterpreter("local",
+ """
+ |import org.apache.spark.sql.functions._
+ |import org.apache.spark.sql.Encoder
+ |import org.apache.spark.sql.expressions.Aggregator
+ |import org.apache.spark.sql.TypedColumn
+ |val simpleSum = new Aggregator[Int, Int, Int] {
+ | def zero: Int = 0 // The initial value.
+ | def reduce(b: Int, a: Int) = b + a // Add an element to the running total
+ | def merge(b1: Int, b2: Int) = b1 + b2 // Merge intermediate values.
+ | def finish(b: Int) = b // Return the final result.
+ |}.toColumn
+ |
+ |val ds = Seq(1, 2, 3, 4).toDS()
+ |ds.select(simpleSum).collect
--- End diff --
I was just thinking you do something like create a file and then make sure that file handle gets used in the closure. If you got something wrong the second time the file is created it will fail.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199090616
**[Test build #53651 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53651/consoleFull)** for PR 11410 at commit [`47e0e39`](https://github.com/apache/spark/commit/47e0e393c907ffa54ac23c9902f5db00dcbca147).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-194108194
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-196675688
**[Test build #53157 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53157/consoleFull)** for PR 11410 at commit [`9be89ff`](https://github.com/apache/spark/commit/9be89ffaa91f8eaef62397db3314d89e0b04fc18).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189613889
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by ScrapCodes <gi...@git.apache.org>.
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/11410#discussion_r54555457
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/OuterScopes.scala ---
@@ -39,4 +41,35 @@ object OuterScopes {
def addOuterScope(outer: AnyRef): Unit = {
outerScopes.putIfAbsent(outer.getClass.getName, outer)
}
+
+ def getOuterScope(innerCls: Class[_]): AnyRef = {
+ assert(innerCls.isMemberClass)
+ val outerCls = innerCls.getDeclaringClass.getName
+ val outer = outerScopes.get(outerCls)
+ if (outer == null) {
+ outerCls match {
+ case REPLClass(line) =>
+ val loader = Utils.getContextOrSparkClassLoader
+ def loadCls(clsName: String): Class[_] = Class.forName(clsName, true, loader)
+
+ val cls1 = loadCls(line + "$read$")
+ val obj1 = cls1.getField("MODULE$").get(null)
+
+ val obj2 = cls1.getMethod("INSTANCE").invoke(obj1)
+ val cls2 = loadCls(line + "$read")
+
+ val obj3 = cls2.getMethod("$iw").invoke(obj2)
+ val cls3 = loadCls(line + "$read$$iw")
+
+ cls3.getMethod("$iw").invoke(obj3)
+
+ case _ => null
+ }
+ } else {
+ outer
+ }
+ }
+
+ // The format of REPL generated wrapper class's name, e.g. `$line12.$read$$iw$$iw`
--- End diff --
Ping @som-snytt and @retronym, Am I correct about the assumption that these wrappers are not likely to change very frequently ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-198242915
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-194108071
**[Test build #52715 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52715/consoleFull)** for PR 11410 at commit [`9be89ff`](https://github.com/apache/spark/commit/9be89ffaa91f8eaef62397db3314d89e0b04fc18).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199088051
**[Test build #53645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53645/consoleFull)** for PR 11410 at commit [`47e0e39`](https://github.com/apache/spark/commit/47e0e393c907ffa54ac23c9902f5db00dcbca147).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199126070
**[Test build #53651 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53651/consoleFull)** for PR 11410 at commit [`47e0e39`](https://github.com/apache/spark/commit/47e0e393c907ffa54ac23c9902f5db00dcbca147).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189655708
**[Test build #52119 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52119/consoleFull)** for PR 11410 at commit [`a0a8424`](https://github.com/apache/spark/commit/a0a8424b33ff147023504ead0168ce05b64310f4).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-198283125
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by retronym <gi...@git.apache.org>.
Github user retronym commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-198330775
So long as you've got some defence from a test case, your current solution seems okay.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-193660924
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199126314
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189813026
**[Test build #52137 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52137/consoleFull)** for PR 11410 at commit [`a9d832b`](https://github.com/apache/spark/commit/a9d832bb9e2f92bf5fb35a7d7283785f4c952304).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` // `INSTANCE()` method to get the single instance of class `$read`. Then call `$iw()``
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189687161
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199090260
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189632314
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/11410
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189613060
cc @marmbrus @liancheng @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/11410#discussion_r54334598
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/OuterScopes.scala ---
@@ -39,4 +41,31 @@ object OuterScopes {
def addOuterScope(outer: AnyRef): Unit = {
outerScopes.putIfAbsent(outer.getClass.getName, outer)
}
+
+ def getOuterScope(innerCls: Class[_]): AnyRef = {
+ assert(innerCls.isMemberClass)
+ val outerCls = innerCls.getDeclaringClass.getName
+ val outer = outerScopes.get(outerCls)
+ if (outer == null) {
+ outerCls match {
+ case REPLClass(line) =>
+ val cls1 = Utils.classForName(line + "$read$")
--- End diff --
Let's add a comment here to explain the REPL use case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189614756
**[Test build #52116 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52116/consoleFull)** for PR 11410 at commit [`a0a8424`](https://github.com/apache/spark/commit/a0a8424b33ff147023504ead0168ce05b64310f4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199157021
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-196642687
**[Test build #53157 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53157/consoleFull)** for PR 11410 at commit [`9be89ff`](https://github.com/apache/spark/commit/9be89ffaa91f8eaef62397db3314d89e0b04fc18).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-193744207
**[Test build #52650 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52650/consoleFull)** for PR 11410 at commit [`8b79215`](https://github.com/apache/spark/commit/8b7921582868a7b9298385bcc11f6f0547969de0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` |createFile();case class TestCaseClass(value: Int)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by retronym <gi...@git.apache.org>.
Github user retronym commented on a diff in the pull request:
https://github.com/apache/spark/pull/11410#discussion_r54557328
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/OuterScopes.scala ---
@@ -39,4 +41,35 @@ object OuterScopes {
def addOuterScope(outer: AnyRef): Unit = {
outerScopes.putIfAbsent(outer.getClass.getName, outer)
}
+
+ def getOuterScope(innerCls: Class[_]): AnyRef = {
+ assert(innerCls.isMemberClass)
+ val outerCls = innerCls.getDeclaringClass.getName
+ val outer = outerScopes.get(outerCls)
+ if (outer == null) {
+ outerCls match {
+ case REPLClass(line) =>
+ val loader = Utils.getContextOrSparkClassLoader
+ def loadCls(clsName: String): Class[_] = Class.forName(clsName, true, loader)
+
+ val cls1 = loadCls(line + "$read$")
+ val obj1 = cls1.getField("MODULE$").get(null)
+
+ val obj2 = cls1.getMethod("INSTANCE").invoke(obj1)
+ val cls2 = loadCls(line + "$read")
+
+ val obj3 = cls2.getMethod("$iw").invoke(obj2)
+ val cls3 = loadCls(line + "$read$$iw")
+
+ cls3.getMethod("$iw").invoke(obj3)
+
+ case _ => null
+ }
+ } else {
+ outer
+ }
+ }
+
+ // The format of REPL generated wrapper class's name, e.g. `$line12.$read$$iw$$iw`
--- End diff --
The assumption is incorrect. Additional nested `iw` names are needed to typecheck when previous lines include imports.
```
⚡ (export X='getClass.getName.mkString("\u200B")'; printf "$X\n$X\nval foo = 0\nimport foo._\n$X\n" | scala -Yrepl-class-based)
Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_71).
Type in expressions for evaluation. Or try :help.
scala> getClass.getName.mkString("")
res0: String = $line3.$read$$iw$$iw
scala> getClass.getName.mkString("")
res1: String = $line4.$read$$iw$$iw
scala> val foo = 0
foo: Int = 0
scala> import foo._
import foo._
scala> getClass.getName.mkString("")
res2: String = $line10.$read$$iw$$iw$$iw$$iw
scala> :quit
```
Re running this with `-Xprint-parser` helps see the structure.
Note that the REPL strips `$iw.` from output. I've inserted zero-width spaces into the strings to be able to see them.
You can disable this feature programmatically:
```
scala> $intp.isettings.unwrapStrings = false
$intp.isettings.unwrapStrings: Boolean = false
scala> getClass.getName
res2: String = $line7.$read$$iw$$iw$
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-193653823
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52645/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-198407456
**[Test build #53541 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53541/consoleFull)** for PR 11410 at commit [`41c2cc5`](https://github.com/apache/spark/commit/41c2cc595e30bdc1237acd3e309050910758ca39).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199157023
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53658/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189813853
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/11410#discussion_r54334612
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/OuterScopes.scala ---
@@ -39,4 +41,35 @@ object OuterScopes {
def addOuterScope(outer: AnyRef): Unit = {
outerScopes.putIfAbsent(outer.getClass.getName, outer)
}
+
+ def getOuterScope(innerCls: Class[_]): AnyRef = {
+ assert(innerCls.isMemberClass)
+ val outerCls = innerCls.getDeclaringClass.getName
+ val outer = outerScopes.get(outerCls)
+ if (outer == null) {
+ outerCls match {
+ case REPLClass(line) =>
+ val loader = Utils.getContextOrSparkClassLoader
+ def loadCls(clsName: String): Class[_] = Class.forName(clsName, true, loader)
+
+ val cls1 = loadCls(line + "$read$")
+ val obj1 = cls1.getField("MODULE$").get(null)
+
+ val obj2 = cls1.getMethod("INSTANCE").invoke(obj1)
+ val cls2 = loadCls(line + "$read")
+
+ val obj3 = cls2.getMethod("$iw").invoke(obj2)
+ val cls3 = loadCls(line + "$read$$iw")
+
+ cls3.getMethod("$iw").invoke(obj3)
+
+ case _ => null
+ }
+ } else {
+ outer
+ }
+ }
+
+ // The format of REPL generated wrapper class's name, e.g. `$line12.$read$$iw$$iw`
--- End diff --
I believe @ScrapCodes is the expert here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-198457435
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-198243532
**[Test build #53520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53520/consoleFull)** for PR 11410 at commit [`9be89ff`](https://github.com/apache/spark/commit/9be89ffaa91f8eaef62397db3314d89e0b04fc18).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199089511
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53645/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-190653743
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52228/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189687163
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52126/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-198282279
**[Test build #53520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53520/consoleFull)** for PR 11410 at commit [`9be89ff`](https://github.com/apache/spark/commit/9be89ffaa91f8eaef62397db3314d89e0b04fc18).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-198240951
ping @retronym again
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-193653822
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-195174175
@retronym Thank you for the review! I'd like to merge this fix. Sounds good?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189613419
**[Test build #52115 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52115/consoleFull)** for PR 11410 at commit [`a2c4617`](https://github.com/apache/spark/commit/a2c461746b10fab4acac5e206a759a9bebbc7c29).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11410#discussion_r54331255
--- Diff: repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala ---
@@ -296,6 +320,31 @@ class ReplSuite extends SparkFunSuite {
}
}
+ test("Datasets agg type-inference") {
+ val output = runInterpreter("local",
+ """
+ |import org.apache.spark.sql.functions._
+ |import org.apache.spark.sql.Encoder
+ |import org.apache.spark.sql.expressions.Aggregator
+ |import org.apache.spark.sql.TypedColumn
+ |/** An `Aggregator` that adds up any numeric type returned by the given function. */
+ |class SumOf[I, N : Numeric](f: I => N) extends
+ | org.apache.spark.sql.expressions.Aggregator[I, N, N] {
--- End diff --
The import doesn't work here, not sure why
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-191101144
**[Test build #52303 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52303/consoleFull)** for PR 11410 at commit [`a0510c4`](https://github.com/apache/spark/commit/a0510c42b6a63468d16f80b5cc81b7367d8c67a2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189813855
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52137/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11410#discussion_r54535121
--- Diff: repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala ---
@@ -254,6 +254,30 @@ class ReplSuite extends SparkFunSuite {
|import sqlContext.implicits._
|case class TestCaseClass(value: Int)
|sc.parallelize(1 to 10).map(x => TestCaseClass(x)).toDF().collect()
+ |
+ |// Test Dataset Serialization in the REPL
+ |Seq(TestCaseClass(1)).toDS().collect()
+ """.stripMargin)
+ assertDoesNotContain("error:", output)
+ assertDoesNotContain("Exception", output)
+ }
+
+ test("Datasets and encoders") {
+ val output = runInterpreter("local",
+ """
+ |import org.apache.spark.sql.functions._
+ |import org.apache.spark.sql.Encoder
+ |import org.apache.spark.sql.expressions.Aggregator
+ |import org.apache.spark.sql.TypedColumn
+ |val simpleSum = new Aggregator[Int, Int, Int] {
+ | def zero: Int = 0 // The initial value.
+ | def reduce(b: Int, a: Int) = b + a // Add an element to the running total
+ | def merge(b1: Int, b2: Int) = b1 + b2 // Merge intermediate values.
+ | def finish(b: Int) = b // Return the final result.
+ |}.toColumn
+ |
+ |val ds = Seq(1, 2, 3, 4).toDS()
+ |ds.select(simpleSum).collect
--- End diff --
I tried to add such a test, but couldn't figure it out. The outer scope is the line wrapper class that generated by scala REPL framework and I'm not sure how to insert side effect into it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199126320
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53651/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11410#discussion_r55781359
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/OuterScopes.scala ---
@@ -39,4 +41,47 @@ object OuterScopes {
def addOuterScope(outer: AnyRef): Unit = {
outerScopes.putIfAbsent(outer.getClass.getName, outer)
}
+
+ def getOuterScope(innerCls: Class[_]): AnyRef = {
+ assert(innerCls.isMemberClass)
+ val outerClassName = innerCls.getDeclaringClass.getName
+ val outer = outerScopes.get(outerClassName)
+ if (outer == null) {
+ outerClassName match {
+ // If the outer class is generated by REPL, users don't need to register it as it has
+ // only one instance and there is a way to retrieve it: get the `$read` object, call the
+ // `INSTANCE()` method to get the single instance of class `$read`. Then call `$iw()`
+ // method multiply times to get the single instance of the inner most `$iw` class.
+ case REPLClass(baseClassName) =>
+ val objClass = Utils.classForName(baseClassName + "$")
+ val objInstance = objClass.getField("MODULE$").get(null)
+ val baseInstance = objClass.getMethod("INSTANCE").invoke(objInstance)
+ val baseClass = Utils.classForName(baseClassName)
+
+ var getter = iwGetter(baseClass)
+ var obj = baseInstance
+ while (getter != null) {
+ obj = getter.invoke(obj)
+ getter = iwGetter(getter.getReturnType)
+ }
+
--- End diff --
hi @retronym , now I loop until there is no `$iw` method, is this looks good to you?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199089498
**[Test build #53645 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53645/consoleFull)** for PR 11410 at commit [`47e0e39`](https://github.com/apache/spark/commit/47e0e393c907ffa54ac23c9902f5db00dcbca147).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199135927
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189660014
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189687073
**[Test build #52126 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52126/consoleFull)** for PR 11410 at commit [`a0a8424`](https://github.com/apache/spark/commit/a0a8424b33ff147023504ead0168ce05b64310f4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199393749
Thanks. Merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11410#discussion_r54672837
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/OuterScopes.scala ---
@@ -39,4 +41,35 @@ object OuterScopes {
def addOuterScope(outer: AnyRef): Unit = {
outerScopes.putIfAbsent(outer.getClass.getName, outer)
}
+
+ def getOuterScope(innerCls: Class[_]): AnyRef = {
+ assert(innerCls.isMemberClass)
+ val outerCls = innerCls.getDeclaringClass.getName
+ val outer = outerScopes.get(outerCls)
+ if (outer == null) {
+ outerCls match {
+ case REPLClass(line) =>
+ val loader = Utils.getContextOrSparkClassLoader
+ def loadCls(clsName: String): Class[_] = Class.forName(clsName, true, loader)
+
+ val cls1 = loadCls(line + "$read$")
+ val obj1 = cls1.getField("MODULE$").get(null)
+
+ val obj2 = cls1.getMethod("INSTANCE").invoke(obj1)
+ val cls2 = loadCls(line + "$read")
+
+ val obj3 = cls2.getMethod("$iw").invoke(obj2)
+ val cls3 = loadCls(line + "$read$$iw")
+
+ cls3.getMethod("$iw").invoke(obj3)
+
+ case _ => null
+ }
+ } else {
+ outer
+ }
+ }
+
+ // The format of REPL generated wrapper class's name, e.g. `$line12.$read$$iw$$iw`
--- End diff --
Thanks for your explanation!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199163066
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199163405
**[Test build #53666 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53666/consoleFull)** for PR 11410 at commit [`47e0e39`](https://github.com/apache/spark/commit/47e0e393c907ffa54ac23c9902f5db00dcbca147).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-191079873
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by retronym <gi...@git.apache.org>.
Github user retronym commented on a diff in the pull request:
https://github.com/apache/spark/pull/11410#discussion_r55785695
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/OuterScopes.scala ---
@@ -39,4 +41,47 @@ object OuterScopes {
def addOuterScope(outer: AnyRef): Unit = {
outerScopes.putIfAbsent(outer.getClass.getName, outer)
}
+
+ def getOuterScope(innerCls: Class[_]): AnyRef = {
+ assert(innerCls.isMemberClass)
+ val outerClassName = innerCls.getDeclaringClass.getName
+ val outer = outerScopes.get(outerClassName)
+ if (outer == null) {
+ outerClassName match {
+ // If the outer class is generated by REPL, users don't need to register it as it has
+ // only one instance and there is a way to retrieve it: get the `$read` object, call the
+ // `INSTANCE()` method to get the single instance of class `$read`. Then call `$iw()`
+ // method multiply times to get the single instance of the inner most `$iw` class.
+ case REPLClass(baseClassName) =>
+ val objClass = Utils.classForName(baseClassName + "$")
+ val objInstance = objClass.getField("MODULE$").get(null)
+ val baseInstance = objClass.getMethod("INSTANCE").invoke(objInstance)
+ val baseClass = Utils.classForName(baseClassName)
+
+ var getter = iwGetter(baseClass)
+ var obj = baseInstance
+ while (getter != null) {
+ obj = getter.invoke(obj)
+ getter = iwGetter(getter.getReturnType)
+ }
+
--- End diff --
Yes, this looks cleaner to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189621791
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-191079878
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52286/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-198283134
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53520/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11410#discussion_r54678025
--- Diff: repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala ---
@@ -296,6 +320,31 @@ class ReplSuite extends SparkFunSuite {
}
}
+ test("Datasets agg type-inference") {
+ val output = runInterpreter("local",
+ """
+ |import org.apache.spark.sql.functions._
+ |import org.apache.spark.sql.Encoder
+ |import org.apache.spark.sql.expressions.Aggregator
+ |import org.apache.spark.sql.TypedColumn
+ |/** An `Aggregator` that adds up any numeric type returned by the given function. */
+ |class SumOf[I, N : Numeric](f: I => N) extends
+ | org.apache.spark.sql.expressions.Aggregator[I, N, N] {
--- End diff --
created https://issues.apache.org/jira/browse/SPARK-13611 for this issue
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189613884
**[Test build #52115 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52115/consoleFull)** for PR 11410 at commit [`a2c4617`](https://github.com/apache/spark/commit/a2c461746b10fab4acac5e206a759a9bebbc7c29).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-199089510
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11410#discussion_r54452439
--- Diff: repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala ---
@@ -254,6 +254,30 @@ class ReplSuite extends SparkFunSuite {
|import sqlContext.implicits._
|case class TestCaseClass(value: Int)
|sc.parallelize(1 to 10).map(x => TestCaseClass(x)).toDF().collect()
+ |
+ |// Test Dataset Serialization in the REPL
+ |Seq(TestCaseClass(1)).toDS().collect()
+ """.stripMargin)
+ assertDoesNotContain("error:", output)
+ assertDoesNotContain("Exception", output)
+ }
+
+ test("Datasets and encoders") {
+ val output = runInterpreter("local",
+ """
+ |import org.apache.spark.sql.functions._
+ |import org.apache.spark.sql.Encoder
+ |import org.apache.spark.sql.expressions.Aggregator
+ |import org.apache.spark.sql.TypedColumn
+ |val simpleSum = new Aggregator[Int, Int, Int] {
+ | def zero: Int = 0 // The initial value.
+ | def reduce(b: Int, a: Int) = b + a // Add an element to the running total
+ | def merge(b1: Int, b2: Int) = b1 + b2 // Merge intermediate values.
+ | def finish(b: Int) = b // Return the final result.
+ |}.toColumn
+ |
+ |val ds = Seq(1, 2, 3, 4).toDS()
+ |ds.select(simpleSum).collect
--- End diff --
I think your solution works, but it might be good to add a test case where the scope that gets captured has a side effect (i.e. create a file so if you double execute the outer scope it will fail the second time).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-189632657
**[Test build #52119 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52119/consoleFull)** for PR 11410 at commit [`a0a8424`](https://github.com/apache/spark/commit/a0a8424b33ff147023504ead0168ce05b64310f4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-191044564
**[Test build #52286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52286/consoleFull)** for PR 11410 at commit [`a0510c4`](https://github.com/apache/spark/commit/a0510c42b6a63468d16f80b5cc81b7367d8c67a2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13456][SQL] fix creating encoders for c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11410#issuecomment-193744962
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52650/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org