You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by drewrobb <gi...@git.apache.org> on 2017/07/29 01:57:51 UTC
[GitHub] spark pull request #18766: [SPARK-8288][SQL] ScalaReflection can use compani...
GitHub user drewrobb opened a pull request:
https://github.com/apache/spark/pull/18766
[SPARK-8288][SQL] ScalaReflection can use companion object constructor
## What changes were proposed in this pull request?
This change fixes a particular scenario where default spark SQL can't encode types that are generated by twitter scrooge. In particular, these types have the pattern of being a trait with a constructor defined only in a companion object, rather than a case class. The situation is described in the original issue in more detail. In this case, the type has no corresponding constructor symbol and causes an exception. This change catches the case where the type has no constructor and looks for an `apply` method on the type's companion object. There might be situations in which this approach would also fail in a new way, but it does work for the specific scrooge example,.
## How was this patch tested?
I've added a single new test that failed prior to my fix. The failure is shown below. The test includes an example type that resembles what scrooge generates. For an full scrooge example, see https://gist.github.com/anonymous/ba13d4b612396ca72725eaa989900314
```
[info] - SPARK-8288 *** FAILED *** (7 milliseconds)
[info] scala.ScalaReflectionException: <none> is not a term
[info] at scala.reflect.api.Symbols$SymbolApi$class.asTerm(Symbols.scala:199)
[info] at scala.reflect.internal.Symbols$SymbolContextApiImpl.asTerm(Symbols.scala:84)
[info] at org.apache.spark.sql.catalyst.ScalaReflection$class.constructParams(ScalaReflection.scala:872)
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/drewrobb/spark SPARK-8288
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18766.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18766
----
commit f1cda4c0556319d829730f709f45104419b7b692
Author: Drew Robb <dr...@gmail.com>
Date: 2017-07-29T01:45:00Z
ScalaReflection can use companion object constructor
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18766: [SPARK-8288][SQL] ScalaReflection can use companion obje...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18766
**[Test build #80032 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80032/testReport)** for PR 18766 at commit [`f1cda4c`](https://github.com/apache/spark/commit/f1cda4c0556319d829730f709f45104419b7b692).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18766: [SPARK-8288][SQL] ScalaReflection can use companion obje...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18766
Is the outcome here that this is an incomplete fix? should it be closed?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18766: [SPARK-8288][SQL] ScalaReflection can use companion obje...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18766
cc @liancheng Should we still support this?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18766: [SPARK-8288][SQL] ScalaReflection can use compani...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/18766
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18766: [SPARK-8288][SQL] ScalaReflection can use companion obje...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18766
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18766: [SPARK-8288][SQL] ScalaReflection can use companion obje...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18766
**[Test build #80032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80032/testReport)** for PR 18766 at commit [`f1cda4c`](https://github.com/apache/spark/commit/f1cda4c0556319d829730f709f45104419b7b692).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18766: [SPARK-8288][SQL] ScalaReflection can use companion obje...
Posted by drewrobb <gi...@git.apache.org>.
Github user drewrobb commented on the issue:
https://github.com/apache/spark/pull/18766
fyi @liancheng this was entirely your suggested fix
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18766: [SPARK-8288][SQL] ScalaReflection can use companion obje...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18766
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80032/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org