You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dongjoon-hyun <gi...@git.apache.org> on 2016/11/25 11:30:51 UTC
[GitHub] spark pull request #16012: [SPARK-17251][SQL] Support `OuterReference` in pr...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/16012
[SPARK-17251][SQL] Support `OuterReference` in projection list of a correlated subquery
## What changes were proposed in this pull request?
Currently, correlated subqueries does not allow `OuterReference` columns in projection lists. This PR aims to support that by making `OuterReference` as `NamedExpression` and extending `pullOutCorrelatedPredicates` into `pullOutCorrelatedProjectionAndPredicates`.
```scala
scala> sql("CREATE TEMPORARY VIEW t1 AS SELECT * FROM VALUES 1, 2 AS t1(a)")
scala> sql("CREATE TEMPORARY VIEW t2 AS SELECT * FROM VALUES 1 AS t2(b)")
scala> sql("SELECT a FROM t1 WHERE a IN (SELECT a FROM t2)").show
java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.OuterReference cannot be cast to org.apache.spark.sql.catalyst.expressions.NamedExpression
```
## How was this patch tested?
Pass the Jenkins test with new test cases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-17251
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16012.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16012
----
commit 58449b317f9a04e5cba2f106afede19dbfde953e
Author: Dongjoon Hyun <do...@apache.org>
Date: 2016-11-24T22:15:04Z
[SPARK-17251][SQL] Support `OuterReference` in projection list of a correlated subquery
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16012: [SPARK-17251][SQL] Support `OuterReference` in pr...
Posted by nsyca <gi...@git.apache.org>.
Github user nsyca commented on a diff in the pull request:
https://github.com/apache/spark/pull/16012#discussion_r89664100
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -989,7 +989,7 @@ class Analyzer(
withPosition(u) {
try {
outer.resolve(nameParts, resolver) match {
- case Some(outerAttr) => OuterReference(outerAttr)
+ case Some(outerAttr) => OuterReference(outerAttr)()
--- End diff --
Another interesting case to consider:
````sql
select ...
from t1
where t1.c1 in (select sum(t1.c2) from t2)
````
If we support correlated columns in SELECT clause, do we build the Aggregate on T2 or T1?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16012: [SPARK-17251][SQL] Support `OuterReference` in projectio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16012
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16012: [SPARK-17251][SQL] Support `OuterReference` in projectio...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/16012
Thank you, @hvanhovell ! I'll fix like that!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16012: [SPARK-17251][SQL] Support `OuterReference` in pr...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16012#discussion_r89615090
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala ---
@@ -356,10 +356,17 @@ case class PrettyAttribute(
* A place holder used to hold a reference that has been resolved to a field outside of the current
* plan. This is used for correlated subqueries.
*/
-case class OuterReference(e: NamedExpression) extends LeafExpression with Unevaluable {
+case class OuterReference(e: NamedExpression)
+ extends LeafExpression with NamedExpression with Unevaluable {
override def dataType: DataType = e.dataType
override def nullable: Boolean = e.nullable
override def prettyName: String = "outer"
+
+ override def name: String = throw new UnsupportedOperationException
--- End diff --
Implement as much of these as possible (delegate to the NamedExpression). It is a resolved expression after all.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16012: [SPARK-17251][SQL] Support `OuterReference` in projectio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16012
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69163/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16012: [SPARK-17251][SQL] Support `OuterReference` in pr...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/16012#discussion_r89656824
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -989,7 +989,7 @@ class Analyzer(
withPosition(u) {
try {
outer.resolve(nameParts, resolver) match {
- case Some(outerAttr) => OuterReference(outerAttr)
+ case Some(outerAttr) => OuterReference(outerAttr)()
--- End diff --
Hmm. Correct. I'll check that again.
BTW, What about the predicates? It felt the predicates are handled the same way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16012: [SPARK-17251][SQL] Support `OuterReference` in pr...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16012#discussion_r89615403
--- Diff: sql/core/src/test/resources/sql-tests/inputs/correlated.sql ---
@@ -0,0 +1,18 @@
+CREATE TEMPORARY VIEW t1 AS SELECT * FROM VALUES 1, 2 AS t1(a);
--- End diff --
Naming: lets call this `subqueries.sql`. Correlation has a different meaning in different contexts
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16012: [SPARK-17251][SQL] Support `OuterReference` in projectio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16012
**[Test build #69163 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69163/consoleFull)** for PR 16012 at commit [`58449b3`](https://github.com/apache/spark/commit/58449b317f9a04e5cba2f106afede19dbfde953e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class OuterReference(e: NamedExpression)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16012: [SPARK-17251][SQL] Support `OuterReference` in pr...
Posted by nsyca <gi...@git.apache.org>.
Github user nsyca commented on a diff in the pull request:
https://github.com/apache/spark/pull/16012#discussion_r89657974
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -989,7 +989,7 @@ class Analyzer(
withPosition(u) {
try {
outer.resolve(nameParts, resolver) match {
- case Some(outerAttr) => OuterReference(outerAttr)
+ case Some(outerAttr) => OuterReference(outerAttr)()
--- End diff --
I have not looked at the code changes closely but got a general idea of what the originally reported problem is. I second @hvanhovell to not support outer reference in a SELECT clause of a subquery in 2.1. Just fix the named expression first.
IN subquery might be okay as it reflects the inner join semantics more or less. NOT IN subquery is converted to a special case of an anti-join with extra logic for the null value.
````sql
select *
from tbl_a
where tbl_a.c1 not in (select tbl_a.c2 from tbl_b)
````
Does the LeftAnti with effectively no join predicate, i.e.,
`(isnull(tbl_a.c1 = tbl_a.c2) || (tbl_a.c1 = tbl_a.c2))`
work correctly today? And if it returns a correct result, is it by design, not by chance?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16012: [SPARK-17251][SQL] Support `OuterReference` in pr...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16012#discussion_r89656204
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -989,7 +989,7 @@ class Analyzer(
withPosition(u) {
try {
outer.resolve(nameParts, resolver) match {
- case Some(outerAttr) => OuterReference(outerAttr)
+ case Some(outerAttr) => OuterReference(outerAttr)()
--- End diff --
I am not sure the analyzer change has the desired effect. This just remove the outer reference from the tree, and this won't work if we use the attribute anywhere in the tree. For example:
```sql
select *
from tbl_a
where id in (select x
from (select tbl_b.id,
tbl_a.id + 1 as x,
tbl_a.id + tbl_b.id as y
from tbl_b)
where y > 0)
```
I think we need to break this down into two steps:
1. Do not support this for now and just fix the named expression. That would be my goal for 2.1.
2. Try to see if we can rewrite the tree in such a way that we can extract the value. That would be my goal for 2.2. I am not sure how well we can make this work. In the end I think we need a dedicated subquery operator.
cc @nsyca what do you think.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16012: [SPARK-17251][SQL] Support `OuterReference` in pr...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/16012#discussion_r89656679
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala ---
@@ -356,10 +356,17 @@ case class PrettyAttribute(
* A place holder used to hold a reference that has been resolved to a field outside of the current
* plan. This is used for correlated subqueries.
*/
-case class OuterReference(e: NamedExpression) extends LeafExpression with Unevaluable {
+case class OuterReference(e: NamedExpression)(
+ val exprId: ExprId = NamedExpression.newExprId)
--- End diff --
Is it okay? I thought it works like 'Alias'. Anyway, no problem. I'll update like that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16012: [SPARK-17251][SQL] Support `OuterReference` in pr...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16012#discussion_r89699139
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -989,7 +989,7 @@ class Analyzer(
withPosition(u) {
try {
outer.resolve(nameParts, resolver) match {
- case Some(outerAttr) => OuterReference(outerAttr)
+ case Some(outerAttr) => OuterReference(outerAttr)()
--- End diff --
The `LEFT ANTI` join should produce the correct result. Unfortunately we push down the `tbl_a.c1 = tbl_a.c2` expression into the `tbl_a` side of the plan. So we need to fix this. I have created https://issues.apache.org/jira/browse/SPARK-18597 to track this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16012: [SPARK-17251][SQL] Support `OuterReference` in pr...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun closed the pull request at:
https://github.com/apache/spark/pull/16012
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16012: [SPARK-17251][SQL] Support `OuterReference` in projectio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16012
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69173/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16012: [SPARK-17251][SQL] Support `OuterReference` in projectio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16012
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16012: [SPARK-17251][SQL] Support `OuterReference` in projectio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16012
**[Test build #69173 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69173/consoleFull)** for PR 16012 at commit [`3de9419`](https://github.com/apache/spark/commit/3de9419a30790020fb4d562625941dbc5e1772d2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16012: [SPARK-17251][SQL] Support `OuterReference` in projectio...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/16012
Hi, @hvanhovell and @nsyca .
I created a JIRA issue for Option 2 with simple and complex example (@hvanhovell 's).
* https://issues.apache.org/jira/browse/SPARK-18603
Now, I'm closing this PR. Thank you for reviews, @hvanhovell and @nsyca .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16012: [SPARK-17251][SQL] Support `OuterReference` in projectio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16012
**[Test build #69173 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69173/consoleFull)** for PR 16012 at commit [`3de9419`](https://github.com/apache/spark/commit/3de9419a30790020fb4d562625941dbc5e1772d2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class OuterReference(e: NamedExpression)(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16012: [SPARK-17251][SQL] Support `OuterReference` in projectio...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16012
**[Test build #69163 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69163/consoleFull)** for PR 16012 at commit [`58449b3`](https://github.com/apache/spark/commit/58449b317f9a04e5cba2f106afede19dbfde953e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16012: [SPARK-17251][SQL] Support `OuterReference` in projectio...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/16012
Thank you for review, @hvanhovell and @nsyca .
I agree with you. We need enough time for this.
So, the option one for 2.1 is spun off into #16015.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16012: [SPARK-17251][SQL] Support `OuterReference` in pr...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16012#discussion_r89655174
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala ---
@@ -356,10 +356,17 @@ case class PrettyAttribute(
* A place holder used to hold a reference that has been resolved to a field outside of the current
* plan. This is used for correlated subqueries.
*/
-case class OuterReference(e: NamedExpression) extends LeafExpression with Unevaluable {
+case class OuterReference(e: NamedExpression)(
+ val exprId: ExprId = NamedExpression.newExprId)
+ extends LeafExpression with NamedExpression with Unevaluable {
override def dataType: DataType = e.dataType
override def nullable: Boolean = e.nullable
override def prettyName: String = "outer"
+
+ override def name: String = e.name
+ override def qualifier: Option[String] = e.qualifier
+ override def toAttribute: Attribute = e.toAttribute
+ override def newInstance(): NamedExpression = OuterReference(e)()
--- End diff --
`OuterReference(e.newInstance())()`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16012: [SPARK-17251][SQL] Support `OuterReference` in pr...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16012#discussion_r89655153
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala ---
@@ -356,10 +356,17 @@ case class PrettyAttribute(
* A place holder used to hold a reference that has been resolved to a field outside of the current
* plan. This is used for correlated subqueries.
*/
-case class OuterReference(e: NamedExpression) extends LeafExpression with Unevaluable {
+case class OuterReference(e: NamedExpression)(
+ val exprId: ExprId = NamedExpression.newExprId)
--- End diff --
Use the `exprId` of the `NamedExpression`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org