You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tianyi <gi...@git.apache.org> on 2014/09/26 08:58:30 UTC
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
GitHub user tianyi opened a pull request:
https://github.com/apache/spark/pull/2542
[SPARK-3688][SQL]LogicalPlan can't resolve column correctlly
we should add some judgements base on the datatype to handle some particular sql descript in https://issues.apache.org/jira/browse/SPARK-3688
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tianyi/spark resolve-issue
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2542.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2542
----
commit ae0187b4029b460777cc9ffbc9f2d3471ee56fa8
Author: tianyi <ti...@asiainfo-linkage.com>
Date: 2014-09-26T06:43:58Z
SPARK-3688 fix logical bug in LogicalPlan and add a test case for this bug
commit 62dc9f501af081e4b358bc12166dfe2e9bee1c4c
Author: tianyi <ti...@asiainfo-linkage.com>
Date: 2014-09-26T06:49:02Z
SPARK-3688 fix some code style
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-58976342
https://www.github.com/apache/spark/pull/2762
This is a PR created by @chenghao-intel
I had tested my PR with it, it works fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-58576044
I think it would be good to move the tests to `HiveResolutionSuite` so we can make sure that we are doing the same thing as Hive. You can probably use `.saveAsTable` to make the porting easier.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-67407944
[Test build #549 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/549/consoleFull) for PR 2542 at commit [`b708fc7`](https://github.com/apache/spark/commit/b708fc7636143562b950fda5fda778e1cd447ae1).
* This patch **does not merge cleanly**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2542#discussion_r18123559
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala ---
@@ -110,9 +110,18 @@ abstract class LogicalPlan extends QueryPlan[LogicalPlan] with Logging {
}
if (resolver(option.name, remainingParts.head)) {
- // Preserve the case of the user's attribute reference.
- (option.withName(remainingParts.head), remainingParts.tail.toList) :: Nil
+ if (remainingParts.length == 1) {
+ // for simple data type
--- End diff --
Not only for simple data types, we can also select a whole `struct` field.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2542#discussion_r18123371
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala ---
@@ -110,9 +110,18 @@ abstract class LogicalPlan extends QueryPlan[LogicalPlan] with Logging {
}
if (resolver(option.name, remainingParts.head)) {
- // Preserve the case of the user's attribute reference.
- (option.withName(remainingParts.head), remainingParts.tail.toList) :: Nil
+ if (remainingParts.length == 1) {
+ // for simple data type
+ (option.withName(remainingParts.head), Nil) :: Nil
--- End diff --
I don't think we need the `withName` call anymore, since now we preserve case of attribute names by leveraging `resolver` to compare names.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-61046262
[Test build #22519 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22519/consoleFull) for PR 2542 at commit [`b708fc7`](https://github.com/apache/spark/commit/b708fc7636143562b950fda5fda778e1cd447ae1).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-61045960
[Test build #22517 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22517/consoleFull) for PR 2542 at commit [`b708fc7`](https://github.com/apache/spark/commit/b708fc7636143562b950fda5fda778e1cd447ae1).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-56942258
@cloud-fan
I think we should follow some orders to resolve a.b
First, if 'a' match a table name or alias, then we recognize a as a table.
if 'a' did not match any table name or alias, then we expect 'a' should be a structType column.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/2542#discussion_r18191704
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala ---
@@ -110,8 +110,20 @@ abstract class LogicalPlan extends QueryPlan[LogicalPlan] with Logging {
}
if (resolver(option.name, remainingParts.head)) {
- // Preserve the case of the user's attribute reference.
- (option.withName(remainingParts.head), remainingParts.tail.toList) :: Nil
+ (option.dataType, remainingParts) match {
+ // No nesting
+ case (_, _ :: Nil) => (option.withName(remainingParts.head), Nil) :: Nil
+ // Points to nested field(s) of a structure
+ case (_: StructType, _ :: tail) => {
+ try {
+ (resolveNesting(tail, option, resolver), tail.last) :: Nil
+ } catch {
+ case _: Throwable => Nil
--- End diff --
@marmbrus , in the resolveNesting function, RuntimeException will be thrown if the nested references not exists in structType.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-58313595
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21445/consoleFull) for PR 2542 at commit [`bfff1b2`](https://github.com/apache/spark/commit/bfff1b264de3a962e9ce43da7f1541ea727d5b9b).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57083428
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/175/consoleFull) for PR 2542 at commit [`e9cd8be`](https://github.com/apache/spark/commit/e9cd8be5b69af54c1de3219cc8f2c0ad1718615a).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57073055
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20925/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-67802647
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57072994
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20926/consoleFull) for PR 2542 at commit [`a018641`](https://github.com/apache/spark/commit/a018641924fd60ebf54c05990a10001cf9a65a0c).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-58412015
Yeah, it would be good to stick with hive semantics for exactly that reason. I assume you can still access the other `x` as `b.a.x`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57074970
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20926/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57075143
**[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/172/consoleFull)** after a configured wait of `120m`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-61046263
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22519/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57076212
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20929/consoleFull) for PR 2542 at commit [`e9cd8be`](https://github.com/apache/spark/commit/e9cd8be5b69af54c1de3219cc8f2c0ad1718615a).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57062370
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57071901
Hi, @marmbrus . The current codes still have some bugs to fix, I talked @liancheng yesterday, I will push a update later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-58614531
@marmbrus I met a problem when I write the test case:
```
createQueryTest("test ambiguousReferences resolved as hive",
"""
|CREATE TABLE t1 (x int);
|CREATE TABLE t2 (a STRUCT<x:string>, x int);
|INSERT OVERWRITE TABLE t1 SELECT 1 FROM src LIMIT 1;
|INSERT OVERWRITE TABLE t2 SELECT named_struct("x","str"),1 FROM src LIMIT 1;
|SELECT a.x FROM t1 a JOIN t2 b ON a.x = b.k;
""".stripMargin)
```
Then I found spark-sql not support `SELECT named_struct`
Do you have any other way to insert data to table `t2` in spark-sql?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57081837
**[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/174/consoleFull)** after a configured wait of `120m`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-67805585
[Test build #24696 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24696/consoleFull) for PR 2542 at commit [`b708fc7`](https://github.com/apache/spark/commit/b708fc7636143562b950fda5fda778e1cd447ae1).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-56926422
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-56940392
Nice catch! I think the problem is: for a table(a string, b string), when we run "select a.b from test a join test b", we have 2 options to resolve `a.b`. One is "table a, column b", another is "table b, column a.b". Here we know that column a don't have a sub column b, so "table b, column a.b" is invalid, the only option is "table a, column b". However, the system didn't know this and report "ambiguousReferences".
But what if we really have a sub column b for column a? Should we report "ambiguousReferences"? @marmbrus What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57062530
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20921/consoleFull) for PR 2542 at commit [`252dbbb`](https://github.com/apache/spark/commit/252dbbbdaab653a94aa784873ac362b4422494e1).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-60865427
[Test build #22417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22417/consoleFull) for PR 2542 at commit [`efc2df3`](https://github.com/apache/spark/commit/efc2df37e0b14759cd064c5b4e7cbcb77a0faeee).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2542#discussion_r18189432
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala ---
@@ -57,6 +57,12 @@ class SQLQuerySuite extends QueryTest {
1)
}
+ test("test particular table alias") {
--- End diff --
It would be good to add some test cases to make sure errors are thrown for some of the ambiguous cases that were brought up in the PR discussion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57135255
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/191/consoleFull) for PR 2542 at commit [`e9cd8be`](https://github.com/apache/spark/commit/e9cd8be5b69af54c1de3219cc8f2c0ad1718615a).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class RandomForestModel(val trees: Array[DecisionTreeModel], val algo: Algo) extends Serializable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57074276
@cloud-fan, I think it is reasonable for return "ambiguous references" in the case you mentioned, because we can't make sure whether 'a' is a table alias or column name. In my last commits, spark will return "ambiguous references" for your case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2542#discussion_r18080938
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala ---
@@ -57,6 +57,12 @@ class SQLQuerySuite extends QueryTest {
1)
}
+ test("test tianyi") {
--- End diff --
Please rename this test...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57079256
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/174/consoleFull) for PR 2542 at commit [`e9cd8be`](https://github.com/apache/spark/commit/e9cd8be5b69af54c1de3219cc8f2c0ad1718615a).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57130258
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/191/consoleFull) for PR 2542 at commit [`e9cd8be`](https://github.com/apache/spark/commit/e9cd8be5b69af54c1de3219cc8f2c0ad1718615a).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2542#discussion_r18673448
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala ---
@@ -94,39 +94,51 @@ abstract class LogicalPlan extends QueryPlan[LogicalPlan] with Logging {
input: Seq[Attribute],
resolver: Resolver): Option[NamedExpression] = {
- val parts = name.split("\\.")
+ val parts = name.split("\\.").toList
+ // Reference to hive SemanticAnalyzer
+ // we should filter some input Attribute if 'name' contains table alias.
--- End diff --
It would be good to include an example of what you mean here. This code is very complex.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57086391
**[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/175/consoleFull)** after a configured wait of `120m`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-60871231
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22417/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-58941699
Hmm, thats unfortunate. You can use `.saveAsTable(...)`, but that is hard to use with `createQueryTest` since that always calls `reset()`. We should probably open a JIRA that `named_struct` is broken.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-63152146
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23398/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-67398969
[Test build #548 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/548/consoleFull) for PR 2542 at commit [`b708fc7`](https://github.com/apache/spark/commit/b708fc7636143562b950fda5fda778e1cd447ae1).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57074968
**[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20926/consoleFull)** after a configured wait of `120m`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-63147249
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57073173
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/172/consoleFull) for PR 2542 at commit [`a018641`](https://github.com/apache/spark/commit/a018641924fd60ebf54c05990a10001cf9a65a0c).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-58310146
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21445/consoleFull) for PR 2542 at commit [`bfff1b2`](https://github.com/apache/spark/commit/bfff1b264de3a962e9ce43da7f1541ea727d5b9b).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-63147391
[Test build #23398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23398/consoleFull) for PR 2542 at commit [`b708fc7`](https://github.com/apache/spark/commit/b708fc7636143562b950fda5fda778e1cd447ae1).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-56952680
@tianyi Let me raise an example. For `table: { a: { c: String }, b: String }`, if we run `select a.b from test a join test b`, your PR will still give 2 options to resolve `a.b`: "table a, column b", "table b, column a.b" as `a` is StructType. Actually we can tell that `a` have a sub column `c`, not `b`, so "table b, column a.b" is invalid. But it's hard to tell whether the column can be resolved in `LogicalPlan#resolve` as column name can be very complicated like `a.b[1].c.d[0]`
I think a perfect solution is keep all the possible resolve path, and go one by one. If more than one path success, then throw an "ambiguousReferences" exception.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-58313597
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21445/Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-58338634
@cloud-fan I had create a test case named "test ambiguousReferences resolved as hive" , the case is almost the same as yours.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57065879
**[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20921/consoleFull)** after a configured wait of `120m`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi closed the pull request at:
https://github.com/apache/spark/pull/2542
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/2542#discussion_r18192244
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala ---
@@ -110,8 +110,20 @@ abstract class LogicalPlan extends QueryPlan[LogicalPlan] with Logging {
}
if (resolver(option.name, remainingParts.head)) {
- // Preserve the case of the user's attribute reference.
- (option.withName(remainingParts.head), remainingParts.tail.toList) :: Nil
+ (option.dataType, remainingParts) match {
+ // No nesting
+ case (_, _ :: Nil) => (option.withName(remainingParts.head), Nil) :: Nil
+ // Points to nested field(s) of a structure
+ case (_: StructType, _ :: tail) => {
+ try {
+ (resolveNesting(tail, option, resolver), tail.last) :: Nil
+ } catch {
+ case _: Throwable => Nil
--- End diff --
I will try to avoid using exceptions by change some code in resolveNesting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-61046528
Hi, @marmbrus @liancheng
I had rebased this PR after [#2762](https://github.com/apache/spark/pull/2762).
Any more comments on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-60871225
[Test build #22417 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22417/consoleFull) for PR 2542 at commit [`efc2df3`](https://github.com/apache/spark/commit/efc2df37e0b14759cd064c5b4e7cbcb77a0faeee).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2542#discussion_r18123585
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala ---
@@ -110,9 +110,18 @@ abstract class LogicalPlan extends QueryPlan[LogicalPlan] with Logging {
}
if (resolver(option.name, remainingParts.head)) {
- // Preserve the case of the user's attribute reference.
- (option.withName(remainingParts.head), remainingParts.tail.toList) :: Nil
+ if (remainingParts.length == 1) {
+ // for simple data type
+ (option.withName(remainingParts.head), Nil) :: Nil
+ } else if (option.dataType.isInstanceOf[StructType]) {
+ // for complex data type
+ (option.withName(remainingParts.head), remainingParts.tail.toList) :: Nil
+ } else {
+ // for simple data type but multiple parts remaining
+ Nil
+ }
--- End diff --
Maybe this?
```scala
(option.dataType, remainingParts) match {
// No nesting
case (_, _ :: Nil) => (option, Nil) :: Nil
// Points to nested field(s) of a structure
case (_: StructType, _ :: tail) => (option, tail) :: Nil
// Invalid
case _ => Nil
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57073139
@liancheng Hmm..I don't have a hive environment for test...
CREATE TABLE t1(x INT);
CREATE TABLE t2(a STRUCT<x: INT>, k INT);
SELECT a.x FROM t1 a JOIN t2 b;
Without this PR, spark sql will report ambiguousReferences, but how hive resolve `a.x`? "table a, column x" or "table b, column a.x"?
Or do we have to add the `ON` statement?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-67421071
[Test build #549 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/549/consoleFull) for PR 2542 at commit [`b708fc7`](https://github.com/apache/spark/commit/b708fc7636143562b950fda5fda778e1cd447ae1).
* This patch **fails PySpark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2542#discussion_r18196493
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala ---
@@ -110,8 +110,20 @@ abstract class LogicalPlan extends QueryPlan[LogicalPlan] with Logging {
}
if (resolver(option.name, remainingParts.head)) {
- // Preserve the case of the user's attribute reference.
- (option.withName(remainingParts.head), remainingParts.tail.toList) :: Nil
+ (option.dataType, remainingParts) match {
+ // No nesting
+ case (_, _ :: Nil) => (option.withName(remainingParts.head), Nil) :: Nil
+ // Points to nested field(s) of a structure
+ case (_: StructType, _ :: tail) => {
+ try {
+ (resolveNesting(tail, option, resolver), tail.last) :: Nil
+ } catch {
+ case _: Throwable => Nil
--- End diff --
Awesome, thank you for working on this :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57056092
@cloud-fan Sorry, made a mistake in the snippet I used, it should be:
```sql
CREATE TABLE t1(x INT);
CREATE TABLE t2(a STRUCT<x: INT>, k INT);
SELECT a.x FROM t1 a JOIN t2 b ON a.x = b.k;
```
And Hive can resolve `a.x` successfully.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57078355
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20929/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-58451999
@marmbrus , could u help me to review these codes? Any suggestion will be appreciated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-67802692
[Test build #24696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24696/consoleFull) for PR 2542 at commit [`b708fc7`](https://github.com/apache/spark/commit/b708fc7636143562b950fda5fda778e1cd447ae1).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-73837719
It's hard to rebase this branch due to so much changes in the latest master branch.
So I had opened a new PR #4524 for this Issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-63152142
[Test build #23398 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23398/consoleFull) for PR 2542 at commit [`b708fc7`](https://github.com/apache/spark/commit/b708fc7636143562b950fda5fda778e1cd447ae1).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-62629348
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57078351
**[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20929/consoleFull)** after a configured wait of `120m`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2542#discussion_r18189290
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala ---
@@ -110,8 +110,20 @@ abstract class LogicalPlan extends QueryPlan[LogicalPlan] with Logging {
}
if (resolver(option.name, remainingParts.head)) {
- // Preserve the case of the user's attribute reference.
- (option.withName(remainingParts.head), remainingParts.tail.toList) :: Nil
+ (option.dataType, remainingParts) match {
--- End diff --
It might be a little clearer to do do something like:
```scala
val optionalNestedReferences = remainingParts.tail
(option.dataType, optionalNestedReferences) match {
....
```
Which I think is equivalent to what you did, but calls out exactly what we are matching on explicitly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2542#discussion_r18189391
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala ---
@@ -110,8 +110,20 @@ abstract class LogicalPlan extends QueryPlan[LogicalPlan] with Logging {
}
if (resolver(option.name, remainingParts.head)) {
- // Preserve the case of the user's attribute reference.
- (option.withName(remainingParts.head), remainingParts.tail.toList) :: Nil
+ (option.dataType, remainingParts) match {
+ // No nesting
+ case (_, _ :: Nil) => (option.withName(remainingParts.head), Nil) :: Nil
+ // Points to nested field(s) of a structure
+ case (_: StructType, _ :: tail) => {
+ try {
+ (resolveNesting(tail, option, resolver), tail.last) :: Nil
+ } catch {
+ case _: Throwable => Nil
--- End diff --
In what cases are exceptions thrown? If possible we should avoid using exceptions for normal program flow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-61043719
[Test build #22519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22519/consoleFull) for PR 2542 at commit [`b708fc7`](https://github.com/apache/spark/commit/b708fc7636143562b950fda5fda778e1cd447ae1).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-61043457
[Test build #22517 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22517/consoleFull) for PR 2542 at commit [`b708fc7`](https://github.com/apache/spark/commit/b708fc7636143562b950fda5fda778e1cd447ae1).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-61045963
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22517/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-61043355
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-67400995
[Test build #548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/548/consoleFull) for PR 2542 at commit [`b708fc7`](https://github.com/apache/spark/commit/b708fc7636143562b950fda5fda778e1cd447ae1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2542#discussion_r18080957
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala ---
@@ -110,9 +110,18 @@ abstract class LogicalPlan extends QueryPlan[LogicalPlan] with Logging {
}
if (resolver(option.name, remainingParts.head)) {
- // Preserve the case of the user's attribute reference.
- (option.withName(remainingParts.head), remainingParts.tail.toList) :: Nil
+ if (remainingParts.length == 1) {
+ //for simple data type
--- End diff --
Add a space after `//`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57065881
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20921/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57039692
@cloud-fan Tried the following snippet with ambiguous references in Hive:
```sql
CREATE TABLE t1(a STRUCT<x: INT>, k INT);
CREATE TABLE t2(x INT);
SELECT a.x FROM t1 a JOIN t2 b ON a.k = b.x;
```
and Hive reports "Invalid column reference 'x'", which I think is reasonable. So it seems that Hive treats struct fields and table columns equally.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-58339330
I think we should follow hive syntax as much as possible, which could save lots of work when people move their application form hive to spark, @marmbrus what's your opinion?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-58338302
I tried this on hive:
CREATE TABLE t1(x INT);
CREATE TABLE t2(a STRUCT<x: INT>, k INT);
SELECT a.x FROM t1 a JOIN t2 b;
And hive can resolve this. However it is reasonable to report "ambiguous references" here, should we follow hive for this case?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-67805590
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24696/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/2542#issuecomment-57074476
@tianyi
CREATE TABLE t1(x INT);
CREATE TABLE t2(a STRUCT<x: INT>, k INT);
SELECT a.x FROM t1 a JOIN t2 b ON a.x = b.k;
But hive can resolve this as @liancheng said. What's the magic here for the `ON` statement?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3688][SQL]LogicalPlan can't resolve col...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2542#discussion_r18189363
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala ---
@@ -110,8 +110,20 @@ abstract class LogicalPlan extends QueryPlan[LogicalPlan] with Logging {
}
if (resolver(option.name, remainingParts.head)) {
- // Preserve the case of the user's attribute reference.
- (option.withName(remainingParts.head), remainingParts.tail.toList) :: Nil
+ (option.dataType, remainingParts) match {
+ // No nesting
+ case (_, _ :: Nil) => (option.withName(remainingParts.head), Nil) :: Nil
+ // Points to nested field(s) of a structure
+ case (_: StructType, _ :: tail) => {
--- End diff --
This could then just be `(_: StructType, nestedFieldReferences) =>` (we don't use `{}` for case statements)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org