You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by kevinyu98 <gi...@git.apache.org> on 2016/05/04 08:50:10 UTC
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
GitHub user kevinyu98 opened a pull request:
https://github.com/apache/spark/pull/12893
[Spark-15051] [SQL] Create a TypedColumn alias
## What changes were proposed in this pull request?
Currently when we try to create an alias against an aggregator TypedColumn, it is using the alias' function from Column, the function will create a column with TypedAggregateExpression, it is unresolved because the inputDeserializer is not defined. But the aggregator function will inject the inputDeserializer back only if it is TypedColumn, so the TypedAggregateExpression will remain unresolved and caused the
problem reported by this jira [15051](https://issues.apache.org/jira/browse/SPARK-15051?jql=project%20%3D%20SPARK).
This PR propose to create a TypedColumn's own alias function which will return TypedColumn , when it is used with aggregator function, the aggregator function will inject the inputDeserializer back .
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
Add test cases in DatasetAggregatorSuite.scala
run the sql related queries against this patch.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kevinyu98/spark spark-15051
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12893.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12893
----
commit 3b44c5978bd44db986621d3e8511e9165b66926b
Author: Kevin Yu <qy...@us.ibm.com>
Date: 2016-04-20T18:06:30Z
adding testcase
commit 18b4a31c687b264b50aa5f5a74455956911f738a
Author: Kevin Yu <qy...@us.ibm.com>
Date: 2016-04-22T21:48:00Z
Merge remote-tracking branch 'upstream/master'
commit 4f4d1c8f2801b1e662304ab2b33351173e71b427
Author: Kevin Yu <qy...@us.ibm.com>
Date: 2016-04-23T16:50:19Z
Merge remote-tracking branch 'upstream/master'
get latest code from upstream
commit f5f0cbed1eb5754c04c36933b374c3b3d2ae4f4e
Author: Kevin Yu <qy...@us.ibm.com>
Date: 2016-04-23T22:20:53Z
Merge remote-tracking branch 'upstream/master'
adding trim characters support
commit d8b2edbd13ee9a4f057bca7dcb0c0940e8e867b8
Author: Kevin Yu <qy...@us.ibm.com>
Date: 2016-04-25T20:24:33Z
Merge remote-tracking branch 'upstream/master'
get latest code for pr12646
commit 196b6c66b0d55232f427c860c0e7c6876c216a67
Author: Kevin Yu <qy...@us.ibm.com>
Date: 2016-04-25T23:45:57Z
Merge remote-tracking branch 'upstream/master'
merge latest code
commit f37a01e005f3e27ae2be056462d6eb6730933ba5
Author: Kevin Yu <qy...@us.ibm.com>
Date: 2016-04-27T14:15:06Z
Merge remote-tracking branch 'upstream/master'
merge upstream/master
commit bb5b01fd3abeea1b03315eccf26762fcc23f80c0
Author: Kevin Yu <qy...@us.ibm.com>
Date: 2016-04-30T23:49:31Z
Merge remote-tracking branch 'upstream/master'
commit 99027fa9cfd3e968bd5dc3808e8af7f8456e1f2d
Author: Kevin Yu <qy...@us.ibm.com>
Date: 2016-05-04T03:51:36Z
fix
commit bde5820a181cf84e0879038ad8c4cebac63c1e24
Author: Kevin Yu <qy...@us.ibm.com>
Date: 2016-05-04T03:52:31Z
Merge remote-tracking branch 'upstream/master'
commit cc8f34006c916d3a5deb50d3def9d6029b514683
Author: Kevin Yu <qy...@us.ibm.com>
Date: 2016-05-04T03:53:53Z
Merge branch 'testing-jira' into spark-15051
commit 0a348415e708464ba101fb0eafa0306c01f23aee
Author: Kevin Yu <qy...@us.ibm.com>
Date: 2016-05-04T07:54:00Z
fixing the typeColumn
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by kevinyu98 <gi...@git.apache.org>.
Github user kevinyu98 commented on the pull request:
https://github.com/apache/spark/pull/12893#issuecomment-217608216
Thank you very much !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12893#issuecomment-217488980
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by kevinyu98 <gi...@git.apache.org>.
Github user kevinyu98 commented on the pull request:
https://github.com/apache/spark/pull/12893#issuecomment-217443570
test please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12893#issuecomment-216794943
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by kevinyu98 <gi...@git.apache.org>.
Github user kevinyu98 commented on the pull request:
https://github.com/apache/spark/pull/12893#issuecomment-216911919
@cloud-fan can you help take a look at this pr? Thanks very much !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by kevinyu98 <gi...@git.apache.org>.
Github user kevinyu98 commented on the pull request:
https://github.com/apache/spark/pull/12893#issuecomment-217329536
@cloud-fan : I have change the codes, can you review it again? Thanks very much.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by kevinyu98 <gi...@git.apache.org>.
Github user kevinyu98 commented on a diff in the pull request:
https://github.com/apache/spark/pull/12893#discussion_r62280265
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ---
@@ -883,7 +895,7 @@ class Column(protected[sql] val expr: Expression) extends Logging {
* @group expr_ops
* @since 1.4.0
*/
- def as(aliases: Seq[String]): Column = withExpr { MultiAlias(expr, aliases) }
+ def as(aliases: Seq[String]): Column = name(aliases)
--- End diff --
I see, I will revert three back, include def as(alias: String, metadata: Metadata): Column. Let me know it is ok. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12893#issuecomment-217465397
**[Test build #57998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57998/consoleFull)** for PR 12893 at commit [`e408fdf`](https://github.com/apache/spark/commit/e408fdf43c207a189f6316a80599e7f54eb832b6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by kevinyu98 <gi...@git.apache.org>.
Github user kevinyu98 commented on a diff in the pull request:
https://github.com/apache/spark/pull/12893#discussion_r62151072
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ---
@@ -68,6 +68,25 @@ class TypedColumn[-T, U](
}
new TypedColumn[T, U](newExpr, encoder)
}
+
+ /** Creates a TypedColumn based on the given expression. */
+ private def withExpr(newExpr: Expression): TypedColumn[T, U] =
+ new TypedColumn[T, U](newExpr, encoder)
+
+ /**
+ * Gives the TypedColumn a name (alias).
+ * If the current TypedColumn has metadata associated with it, this metadata will be propagated
+ * to the new column.
+ *
+ * @group expr_ops
+ * @since 2.0.0
+ */
+ override def as(alias: String): TypedColumn[T, U] = withExpr {
--- End diff --
@rxin @cloud-fan : Thanks very much. I have made the changes based on your comments. Can you help check?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12893#discussion_r62140732
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ---
@@ -68,6 +68,25 @@ class TypedColumn[-T, U](
}
new TypedColumn[T, U](newExpr, encoder)
}
+
+ /** Creates a TypedColumn based on the given expression. */
+ private def withExpr(newExpr: Expression): TypedColumn[T, U] =
+ new TypedColumn[T, U](newExpr, encoder)
+
+ /**
+ * Gives the TypedColumn a name (alias).
+ * If the current TypedColumn has metadata associated with it, this metadata will be propagated
+ * to the new column.
+ *
+ * @group expr_ops
+ * @since 2.0.0
+ */
+ override def as(alias: String): TypedColumn[T, U] = withExpr {
--- End diff --
and it can be `new TypedColumn[T, U](super.name(alias).expr, encoder)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by kevinyu98 <gi...@git.apache.org>.
Github user kevinyu98 commented on the pull request:
https://github.com/apache/spark/pull/12893#issuecomment-217286258
Hello Wenchen: I have gone through all the alias method in Column, delegate to the the "name" method. Can you help check?
BTW, I am just curious why we want to delegate to the "name" method. Is the "name" method intended to be used by the future callers? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/12893#issuecomment-217463479
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by kevinyu98 <gi...@git.apache.org>.
Github user kevinyu98 commented on a diff in the pull request:
https://github.com/apache/spark/pull/12893#discussion_r62203218
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ---
@@ -68,6 +68,18 @@ class TypedColumn[-T, U](
}
new TypedColumn[T, U](newExpr, encoder)
}
+
+ /**
+ * Gives the TypedColumn a name (alias).
+ * If the current TypedColumn has metadata associated with it, this metadata will be propagated
+ * to the new column.
+ *
+ * @group expr_ops
+ * @since 2.0.0
+ */
+ override def name(alias: String): TypedColumn[T, U] =
--- End diff --
Sure, I will look into . Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12893#discussion_r62136859
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ---
@@ -68,6 +68,25 @@ class TypedColumn[-T, U](
}
new TypedColumn[T, U](newExpr, encoder)
}
+
+ /** Creates a TypedColumn based on the given expression. */
+ private def withExpr(newExpr: Expression): TypedColumn[T, U] =
--- End diff --
we don't need this i think? it's used only once.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/12893#issuecomment-217602652
thanks, merging to master and 2.0!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by kevinyu98 <gi...@git.apache.org>.
Github user kevinyu98 commented on a diff in the pull request:
https://github.com/apache/spark/pull/12893#discussion_r62150990
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ---
@@ -68,6 +68,25 @@ class TypedColumn[-T, U](
}
new TypedColumn[T, U](newExpr, encoder)
}
+
+ /** Creates a TypedColumn based on the given expression. */
+ private def withExpr(newExpr: Expression): TypedColumn[T, U] =
--- End diff --
Sure, I will remove it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12893#issuecomment-217488726
**[Test build #57998 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57998/consoleFull)** for PR 12893 at commit [`e408fdf`](https://github.com/apache/spark/commit/e408fdf43c207a189f6316a80599e7f54eb832b6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by kevinyu98 <gi...@git.apache.org>.
Github user kevinyu98 commented on the pull request:
https://github.com/apache/spark/pull/12893#issuecomment-217083169
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12893#issuecomment-217488982
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57998/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/12893#issuecomment-217330185
LGTM, pending jenkins
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12893#discussion_r62279165
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ---
@@ -883,7 +895,7 @@ class Column(protected[sql] val expr: Expression) extends Logging {
* @group expr_ops
* @since 1.4.0
*/
- def as(aliases: Seq[String]): Column = withExpr { MultiAlias(expr, aliases) }
+ def as(aliases: Seq[String]): Column = name(aliases)
--- End diff --
Sorry I didn't make it clearly: delegate to `def name(alias: String)` if we can. For this one, we don't need to touch it as it doesn't make sense to call in on `TypedColumn`.
The reason I want to make as more as possible alias methods delegate to `def name(alias: String)` is: we don't need to override all of them in `TypedColumn`, but just the `name`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12893
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12893#discussion_r62140703
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ---
@@ -68,6 +68,25 @@ class TypedColumn[-T, U](
}
new TypedColumn[T, U](newExpr, encoder)
}
+
+ /** Creates a TypedColumn based on the given expression. */
+ private def withExpr(newExpr: Expression): TypedColumn[T, U] =
+ new TypedColumn[T, U](newExpr, encoder)
+
+ /**
+ * Gives the TypedColumn a name (alias).
+ * If the current TypedColumn has metadata associated with it, this metadata will be propagated
+ * to the new column.
+ *
+ * @group expr_ops
+ * @since 2.0.0
+ */
+ override def as(alias: String): TypedColumn[T, U] = withExpr {
--- End diff --
We should override `name`, which all other alias methods delegate to.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-15051] [SQL] Create a TypedColumn alias...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12893#discussion_r62158438
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ---
@@ -68,6 +68,18 @@ class TypedColumn[-T, U](
}
new TypedColumn[T, U](newExpr, encoder)
}
+
+ /**
+ * Gives the TypedColumn a name (alias).
+ * If the current TypedColumn has metadata associated with it, this metadata will be propagated
+ * to the new column.
+ *
+ * @group expr_ops
+ * @since 2.0.0
+ */
+ override def name(alias: String): TypedColumn[T, U] =
--- End diff --
looks like `def as(alias: Symbol): Column` don't delegate to `name`, we should change it.
Can you go through all alias methods of `Column` and make sure they all delegate to `name` or re-implemented in `TypedColumn`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org