You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2015/11/16 09:31:41 UTC
[GitHub] spark pull request: [SPARK-11754][SQL] consolidate `ExpressionEnco...
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/9729
[SPARK-11754][SQL] consolidate `ExpressionEncoder.tuple` and `Encoders.tuple`
These 2 are very similar, we can consolidate them into one.
Also add tests for it and fix a bug.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark tuple
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9729.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9729
----
commit ae3d592965d8fb34b9458e24f3e53da6915b6d1b
Author: Wenchen Fan <we...@databricks.com>
Date: 2015-11-16T08:19:42Z
consolidate `ExpressionEncoder.tuple` and `Encoders.tuple`
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11754][SQL] consolidate `ExpressionEnco...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/9729#issuecomment-156955343
cc @marmbrus
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11754][SQL] consolidate `ExpressionEnco...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9729#issuecomment-156987643
**[Test build #45988 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45988/consoleFull)** for PR 9729 at commit [`ae3d592`](https://github.com/apache/spark/commit/ae3d592965d8fb34b9458e24f3e53da6915b6d1b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11754][SQL] consolidate `ExpressionEnco...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9729#issuecomment-156988038
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11754][SQL] consolidate `ExpressionEnco...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9729#issuecomment-156988041
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45988/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11754][SQL] consolidate `ExpressionEnco...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/9729#discussion_r44898288
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala ---
@@ -67,47 +67,77 @@ object ExpressionEncoder {
def tuple(encoders: Seq[ExpressionEncoder[_]]): ExpressionEncoder[_] = {
encoders.foreach(_.assertUnresolved())
- val schema =
- StructType(
- encoders.zipWithIndex.map {
- case (e, i) => StructField(s"_${i + 1}", if (e.flat) e.schema.head.dataType else e.schema)
- })
+ val schema = StructType(encoders.zipWithIndex.map {
+ case (e, i) => StructField(s"_${i + 1}", if (e.flat) e.schema.head.dataType else e.schema)
+ })
+
val cls = Utils.getContextOrSparkClassLoader.loadClass(s"scala.Tuple${encoders.size}")
- // Rebind the encoders to the nested schema.
- val newConstructExpressions = encoders.zipWithIndex.map {
- case (e, i) if !e.flat => e.nested(i).fromRowExpression
- case (e, i) => e.shift(i).fromRowExpression
--- End diff --
This is not right. `ExpressionEncoder.shift` update `ordinal` of `BoundReference` with a delta, but here the `i` **is** the `ordinal` we wanna update, not a delta.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11754][SQL] consolidate `ExpressionEnco...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9729#issuecomment-156956597
**[Test build #45988 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45988/consoleFull)** for PR 9729 at commit [`ae3d592`](https://github.com/apache/spark/commit/ae3d592965d8fb34b9458e24f3e53da6915b6d1b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org