You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by marmbrus <gi...@git.apache.org> on 2014/09/13 21:18:44 UTC
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
GitHub user marmbrus opened a pull request:
https://github.com/apache/spark/pull/2382
[SPARK-3414][SQL] Replace LowerCaseSchema with Resolver
_This PR is a follow up to #2293 (and to a lesser extent #2262 #2334)._
In #2293 the catalog was changed to store analyzed logical plans instead of unresolved ones. While this change fixed the reported bug (which was caused by yet another instance of us forgetting to put in a `LowerCaseSchema` operator) it had the consequence of breaking assumptions made by `MultiInstanceRelation`. Specifically, we can't replace swap out leaf operators in a tree without rewriting changed expression ids (which happens when you self join the same RDD that has been registered as a temp table).
In this PR, I instead remove the need to insert `LowerCaseSchema` operators at all, and instead move the concern of matching up identifiers completely into analysis. Doing so allows the test cases from both #2293 and #2262 to pass at the same time (and likely fixes a slew of other "unknown unknown" bugs).
While it is rolled back in this PR, storing the analyzed plan might actually be a good idea. For instance, it is kind of confusing if you register a temporary table, change the case sensitivity of resolution and now you can't query that table anymore. This can be addressed in a follow up PR.
Follow-ups:
- Configurable case sensitivity
- Consider storing analyzed plans for temp tables
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/marmbrus/spark lowercase
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2382.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2382
----
commit 19a61b93b2d3f835d1f1e286b2c91cfcf16371f8
Author: Michael Armbrust <mi...@databricks.com>
Date: 2014-08-27T19:43:38Z
Decrease partitions when testing
commit 76b3e04a7140fb4e06bc51827341659282bff2f8
Author: Michael Armbrust <mi...@databricks.com>
Date: 2014-08-27T20:40:48Z
increase test parallelism
commit fd7b671267687d46d4b7304cd5cdd44c5c926b2f
Author: Michael Armbrust <mi...@databricks.com>
Date: 2014-09-09T01:58:28Z
Make parquet tests less order dependent
commit dc7cb6ea81f5cd1da2247a43e8b4f81e78b52715
Author: Michael Armbrust <mi...@databricks.com>
Date: 2014-09-10T20:59:40Z
more test fixes
commit 9188b59b0b741376608b91dd71afdd3b90ac07f9
Author: Michael Armbrust <mi...@databricks.com>
Date: 2014-09-13T05:55:19Z
Merge branch 'shufflePartitions' into lowercase
commit c2f2ec8b5967b3e69efee4ba054afb96320ac673
Author: Michael Armbrust <mi...@databricks.com>
Date: 2014-09-13T19:00:29Z
Replace LowerCaseSchema with Resolver.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2382#discussion_r17578483
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/package.scala ---
@@ -22,4 +22,9 @@ package org.apache.spark.sql.catalyst
* Analysis consists of translating [[UnresolvedAttribute]]s and [[UnresolvedRelation]]s
* into fully typed objects using information in a schema [[Catalog]].
*/
-package object analysis
+package object analysis {
+ type Resolver = (String, String) => Boolean
+
+ val caseInsensitiveResolution = (a: String, b: String) => a.toLowerCase == b.toLowerCase
--- End diff --
Maybe `a.equalIgnoreCase(b)`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2382#discussion_r17578145
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala ---
@@ -96,8 +101,17 @@ abstract class LogicalPlan extends QueryPlan[LogicalPlan] {
val options = input.flatMap { option =>
// If the first part of the desired name matches a qualifier for this possible match, drop it.
val remainingParts =
- if (option.qualifiers.contains(parts.head) && parts.size > 1) parts.drop(1) else parts
- if (option.name == remainingParts.head) (option, remainingParts.tail.toList) :: Nil else Nil
+ if (option.qualifiers.filter(resolver(_, parts.head)).nonEmpty && parts.size > 1) {
--- End diff --
`find` would be better than `filter` here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2382#issuecomment-55504038
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20275/consoleFull) for PR 2382 at commit [`5b93711`](https://github.com/apache/spark/commit/5b937113ee4a6a071fbc770077ba3f9ba7693b45).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2382#issuecomment-55504003
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20275/consoleFull) for PR 2382 at commit [`5b93711`](https://github.com/apache/spark/commit/5b937113ee4a6a071fbc770077ba3f9ba7693b45).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2382#issuecomment-55503955
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/88/consoleFull) for PR 2382 at commit [`c2f2ec8`](https://github.com/apache/spark/commit/c2f2ec8b5967b3e69efee4ba054afb96320ac673).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2382#issuecomment-55503862
/cc @ericl
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2382#discussion_r18065885
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala ---
@@ -96,21 +103,69 @@ abstract class LogicalPlan extends QueryPlan[LogicalPlan] {
val options = input.flatMap { option =>
// If the first part of the desired name matches a qualifier for this possible match, drop it.
val remainingParts =
- if (option.qualifiers.contains(parts.head) && parts.size > 1) parts.drop(1) else parts
- if (option.name == remainingParts.head) (option, remainingParts.tail.toList) :: Nil else Nil
+ if (option.qualifiers.find(resolver(_, parts.head)).nonEmpty && parts.size > 1) {
+ parts.drop(1)
+ } else {
+ parts
+ }
+
+ if (resolver(option.name, remainingParts.head)) {
+ // Preserve the case of the user's attribute reference.
+ (option.withName(remainingParts.head), remainingParts.tail.toList) :: Nil
+ } else {
+ Nil
+ }
}
options.distinct match {
- case Seq((a, Nil)) => Some(a) // One match, no nested fields, use it.
+ // One match, no nested fields, use it.
+ case Seq((a, Nil)) => Some(a)
+
// One match, but we also need to extract the requested nested field.
case Seq((a, nestedFields)) =>
- Some(Alias(nestedFields.foldLeft(a: Expression)(GetField), nestedFields.last)())
- case Seq() => None // No matches.
+ val aliased =
+ Alias(
+ resolveNesting(nestedFields, a, resolver),
+ nestedFields.last)() // Preserve the case of the user's field access.
+ Some(aliased)
+
+ // No matches.
+ case Seq() =>
+ logTrace(s"Could not find $name in ${input.mkString(", ")}")
+ None
+
+ // More than one match.
case ambiguousReferences =>
throw new TreeNodeException(
this, s"Ambiguous references to $name: ${ambiguousReferences.mkString(",")}")
}
}
+
+ /**
+ * Given a list of successive nested field accesses, and a based expression, attempt to resolve
+ * the actual field lookups on this expression.
+ */
+ private def resolveNesting(
+ nestedFields: List[String],
+ expression: Expression,
+ resolver: Resolver): Expression = {
+
+ (nestedFields, expression.dataType) match {
+ case (Nil, _) => expression
+ case (requestedField :: rest, StructType(fields)) =>
+ val actualField = fields.filter(f => resolver(f.name, requestedField))
--- End diff --
Hmm, good point. Right now you can't make a SQLContext case insensitive, but when you can this will be problem. Maybe you should note this on [SPARK-3617](https://issues.apache.org/jira/browse/SPARK-3617)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2382#discussion_r17620453
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/package.scala ---
@@ -22,4 +22,9 @@ package org.apache.spark.sql.catalyst
* Analysis consists of translating [[UnresolvedAttribute]]s and [[UnresolvedRelation]]s
* into fully typed objects using information in a schema [[Catalog]].
*/
-package object analysis
+package object analysis {
+ type Resolver = (String, String) => Boolean
--- End diff --
I think this will actually end up providing more general resolution functionality in the long term. I've added some scala doc for clarity though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2382#discussion_r17620241
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveResolutionSuite.scala ---
@@ -57,13 +57,14 @@ class HiveResolutionSuite extends HiveComparisonTest {
.registerTempTable("caseSensitivityTest")
sql("SELECT a, b, A, B, n.a, n.b, n.A, n.B FROM caseSensitivityTest")
+ }
- println(sql("SELECT * FROM casesensitivitytest one JOIN casesensitivitytest two ON one.a = two.a").queryExecution)
-
- sql("SELECT * FROM casesensitivitytest one JOIN casesensitivitytest two ON one.a = two.a").collect()
-
- // TODO: sql("SELECT * FROM casesensitivitytest a JOIN casesensitivitytest b ON a.a = b.a")
+ ignore("case insensitivity with scala reflection joins") {
--- End diff --
Yeah, I only changed it to an explicit ignore instead of being commented out. We need to decided if this is allowed or not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2382#issuecomment-56115401
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20557/consoleFull) for PR 2382 at commit [`01cc29a`](https://github.com/apache/spark/commit/01cc29afffa391a1df7323a22b80598b0dfbb0fc).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2382#issuecomment-56115851
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20559/consoleFull) for PR 2382 at commit [`c21171e`](https://github.com/apache/spark/commit/c21171ecd67835dab9441ed48e0f6b76bbbf52ad).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/2382#discussion_r18017841
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala ---
@@ -96,21 +103,69 @@ abstract class LogicalPlan extends QueryPlan[LogicalPlan] {
val options = input.flatMap { option =>
// If the first part of the desired name matches a qualifier for this possible match, drop it.
val remainingParts =
- if (option.qualifiers.contains(parts.head) && parts.size > 1) parts.drop(1) else parts
- if (option.name == remainingParts.head) (option, remainingParts.tail.toList) :: Nil else Nil
+ if (option.qualifiers.find(resolver(_, parts.head)).nonEmpty && parts.size > 1) {
+ parts.drop(1)
+ } else {
+ parts
+ }
+
+ if (resolver(option.name, remainingParts.head)) {
+ // Preserve the case of the user's attribute reference.
+ (option.withName(remainingParts.head), remainingParts.tail.toList) :: Nil
+ } else {
+ Nil
+ }
}
options.distinct match {
- case Seq((a, Nil)) => Some(a) // One match, no nested fields, use it.
+ // One match, no nested fields, use it.
+ case Seq((a, Nil)) => Some(a)
+
// One match, but we also need to extract the requested nested field.
case Seq((a, nestedFields)) =>
- Some(Alias(nestedFields.foldLeft(a: Expression)(GetField), nestedFields.last)())
- case Seq() => None // No matches.
+ val aliased =
+ Alias(
+ resolveNesting(nestedFields, a, resolver),
+ nestedFields.last)() // Preserve the case of the user's field access.
+ Some(aliased)
+
+ // No matches.
+ case Seq() =>
+ logTrace(s"Could not find $name in ${input.mkString(", ")}")
+ None
+
+ // More than one match.
case ambiguousReferences =>
throw new TreeNodeException(
this, s"Ambiguous references to $name: ${ambiguousReferences.mkString(",")}")
}
}
+
+ /**
+ * Given a list of successive nested field accesses, and a based expression, attempt to resolve
+ * the actual field lookups on this expression.
+ */
+ private def resolveNesting(
+ nestedFields: List[String],
+ expression: Expression,
+ resolver: Resolver): Expression = {
+
+ (nestedFields, expression.dataType) match {
+ case (Nil, _) => expression
+ case (requestedField :: rest, StructType(fields)) =>
+ val actualField = fields.filter(f => resolver(f.name, requestedField))
--- End diff --
There is a problem here. Currently `a.b[1].c.d` will be parsed as `GetField(GetField(GetItem(Unresolved("a.b"), 1), "c"), "d")` , so the case-sensitive-check only happens when resolve `Unresolved("a.b")` to `GetField(Attribute("a"), "b")`. Something like "SELECT a[0].A.A from nested" will fail for hql on case-sensitive-check.
I think we should also do this check in `GetField`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2382#discussion_r18066083
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala ---
@@ -96,21 +103,69 @@ abstract class LogicalPlan extends QueryPlan[LogicalPlan] {
val options = input.flatMap { option =>
// If the first part of the desired name matches a qualifier for this possible match, drop it.
val remainingParts =
- if (option.qualifiers.contains(parts.head) && parts.size > 1) parts.drop(1) else parts
- if (option.name == remainingParts.head) (option, remainingParts.tail.toList) :: Nil else Nil
+ if (option.qualifiers.find(resolver(_, parts.head)).nonEmpty && parts.size > 1) {
+ parts.drop(1)
+ } else {
+ parts
+ }
+
+ if (resolver(option.name, remainingParts.head)) {
+ // Preserve the case of the user's attribute reference.
+ (option.withName(remainingParts.head), remainingParts.tail.toList) :: Nil
+ } else {
+ Nil
+ }
}
options.distinct match {
- case Seq((a, Nil)) => Some(a) // One match, no nested fields, use it.
+ // One match, no nested fields, use it.
+ case Seq((a, Nil)) => Some(a)
+
// One match, but we also need to extract the requested nested field.
case Seq((a, nestedFields)) =>
- Some(Alias(nestedFields.foldLeft(a: Expression)(GetField), nestedFields.last)())
- case Seq() => None // No matches.
+ val aliased =
+ Alias(
+ resolveNesting(nestedFields, a, resolver),
+ nestedFields.last)() // Preserve the case of the user's field access.
+ Some(aliased)
+
+ // No matches.
+ case Seq() =>
+ logTrace(s"Could not find $name in ${input.mkString(", ")}")
+ None
+
+ // More than one match.
case ambiguousReferences =>
throw new TreeNodeException(
this, s"Ambiguous references to $name: ${ambiguousReferences.mkString(",")}")
}
}
+
+ /**
+ * Given a list of successive nested field accesses, and a based expression, attempt to resolve
+ * the actual field lookups on this expression.
+ */
+ private def resolveNesting(
+ nestedFields: List[String],
+ expression: Expression,
+ resolver: Resolver): Expression = {
+
+ (nestedFields, expression.dataType) match {
+ case (Nil, _) => expression
+ case (requestedField :: rest, StructType(fields)) =>
+ val actualField = fields.filter(f => resolver(f.name, requestedField))
--- End diff --
Oh wait, sorry... Is that how the HiveQL parser will do it too? I'm not a huge fan of moving resolution logic into the expressions. What about a rule that only ran in case insensitive mode that fixes unresolved GetFields?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2382#issuecomment-55503844
/cc @liancheng
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/2382#discussion_r18078535
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala ---
@@ -96,21 +103,69 @@ abstract class LogicalPlan extends QueryPlan[LogicalPlan] {
val options = input.flatMap { option =>
// If the first part of the desired name matches a qualifier for this possible match, drop it.
val remainingParts =
- if (option.qualifiers.contains(parts.head) && parts.size > 1) parts.drop(1) else parts
- if (option.name == remainingParts.head) (option, remainingParts.tail.toList) :: Nil else Nil
+ if (option.qualifiers.find(resolver(_, parts.head)).nonEmpty && parts.size > 1) {
+ parts.drop(1)
+ } else {
+ parts
+ }
+
+ if (resolver(option.name, remainingParts.head)) {
+ // Preserve the case of the user's attribute reference.
+ (option.withName(remainingParts.head), remainingParts.tail.toList) :: Nil
+ } else {
+ Nil
+ }
}
options.distinct match {
- case Seq((a, Nil)) => Some(a) // One match, no nested fields, use it.
+ // One match, no nested fields, use it.
+ case Seq((a, Nil)) => Some(a)
+
// One match, but we also need to extract the requested nested field.
case Seq((a, nestedFields)) =>
- Some(Alias(nestedFields.foldLeft(a: Expression)(GetField), nestedFields.last)())
- case Seq() => None // No matches.
+ val aliased =
+ Alias(
+ resolveNesting(nestedFields, a, resolver),
+ nestedFields.last)() // Preserve the case of the user's field access.
+ Some(aliased)
+
+ // No matches.
+ case Seq() =>
+ logTrace(s"Could not find $name in ${input.mkString(", ")}")
+ None
+
+ // More than one match.
case ambiguousReferences =>
throw new TreeNodeException(
this, s"Ambiguous references to $name: ${ambiguousReferences.mkString(",")}")
}
}
+
+ /**
+ * Given a list of successive nested field accesses, and a based expression, attempt to resolve
+ * the actual field lookups on this expression.
+ */
+ private def resolveNesting(
+ nestedFields: List[String],
+ expression: Expression,
+ resolver: Resolver): Expression = {
+
+ (nestedFields, expression.dataType) match {
+ case (Nil, _) => expression
+ case (requestedField :: rest, StructType(fields)) =>
+ val actualField = fields.filter(f => resolver(f.name, requestedField))
--- End diff --
Yes, this bug exists in HiveQL. I have opened a PR to fix this(adding a rule to fix unresolved `GetField`s).https://github.com/apache/spark/pull/2543 Need your comments :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2382#discussion_r17622093
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveResolutionSuite.scala ---
@@ -57,13 +57,14 @@ class HiveResolutionSuite extends HiveComparisonTest {
.registerTempTable("caseSensitivityTest")
sql("SELECT a, b, A, B, n.a, n.b, n.A, n.B FROM caseSensitivityTest")
+ }
- println(sql("SELECT * FROM casesensitivitytest one JOIN casesensitivitytest two ON one.a = two.a").queryExecution)
-
- sql("SELECT * FROM casesensitivitytest one JOIN casesensitivitytest two ON one.a = two.a").collect()
-
- // TODO: sql("SELECT * FROM casesensitivitytest a JOIN casesensitivitytest b ON a.a = b.a")
+ ignore("case insensitivity with scala reflection joins") {
--- End diff --
Hive 0.12 actually supports this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/2382#discussion_r17531193
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/package.scala ---
@@ -22,4 +22,9 @@ package org.apache.spark.sql.catalyst
* Analysis consists of translating [[UnresolvedAttribute]]s and [[UnresolvedRelation]]s
* into fully typed objects using information in a schema [[Catalog]].
*/
-package object analysis
+package object analysis {
+ type Resolver = (String, String) => Boolean
--- End diff --
`Resolver` probably a general name, can we use a more precise name for this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2382#issuecomment-55685249
LGTM except some minor issues mentioned in the comments :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2382#discussion_r17622161
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveResolutionSuite.scala ---
@@ -57,13 +57,14 @@ class HiveResolutionSuite extends HiveComparisonTest {
.registerTempTable("caseSensitivityTest")
sql("SELECT a, b, A, B, n.a, n.b, n.A, n.B FROM caseSensitivityTest")
+ }
- println(sql("SELECT * FROM casesensitivitytest one JOIN casesensitivitytest two ON one.a = two.a").queryExecution)
-
- sql("SELECT * FROM casesensitivitytest one JOIN casesensitivitytest two ON one.a = two.a").collect()
-
- // TODO: sql("SELECT * FROM casesensitivitytest a JOIN casesensitivitytest b ON a.a = b.a")
+ ignore("case insensitivity with scala reflection joins") {
--- End diff --
Hmm what is the exception that is thrown, will #2209 fix this? Otherwise we should open a JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2382#issuecomment-55507264
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20276/consoleFull) for PR 2382 at commit [`219805a`](https://github.com/apache/spark/commit/219805a2003f1f56cbf3cde37697a08ac444c124).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/2382
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2382#discussion_r17620503
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala ---
@@ -98,12 +98,12 @@ case class Star(
override def withNullability(newNullability: Boolean) = this
override def withQualifiers(newQualifiers: Seq[String]) = this
- def expand(input: Seq[Attribute]): Seq[NamedExpression] = {
+ def expand(input: Seq[Attribute], resolver: Resolver): Seq[NamedExpression] = {
val expandedAttributes: Seq[Attribute] = table match {
// If there is no table specified, use all input attributes.
case None => input
// If there is a table, pick out attributes that are part of this table.
- case Some(t) => input.filter(_.qualifiers contains t)
+ case Some(t) => input.filter(_.qualifiers.filter(resolver(_,t)).nonEmpty)
--- End diff --
Fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2382#issuecomment-55507198
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/89/consoleFull) for PR 2382 at commit [`5b93711`](https://github.com/apache/spark/commit/5b937113ee4a6a071fbc770077ba3f9ba7693b45).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class JavaSparkContext(val sc: SparkContext)`
* `class TaskCompletionListenerException(errorMessages: Seq[String]) extends Exception `
* ` class Dummy(object):`
* `class JavaStreamingContext(val ssc: StreamingContext) extends Closeable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2382#issuecomment-55504293
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/89/consoleFull) for PR 2382 at commit [`5b93711`](https://github.com/apache/spark/commit/5b937113ee4a6a071fbc770077ba3f9ba7693b45).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2382#issuecomment-56120103
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20557/consoleFull) for PR 2382 at commit [`01cc29a`](https://github.com/apache/spark/commit/01cc29afffa391a1df7323a22b80598b0dfbb0fc).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `abstract class LogicalPlan extends QueryPlan[LogicalPlan] with Logging `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2382#discussion_r17578267
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveResolutionSuite.scala ---
@@ -57,13 +57,14 @@ class HiveResolutionSuite extends HiveComparisonTest {
.registerTempTable("caseSensitivityTest")
sql("SELECT a, b, A, B, n.a, n.b, n.A, n.B FROM caseSensitivityTest")
+ }
- println(sql("SELECT * FROM casesensitivitytest one JOIN casesensitivitytest two ON one.a = two.a").queryExecution)
-
- sql("SELECT * FROM casesensitivitytest one JOIN casesensitivitytest two ON one.a = two.a").collect()
-
- // TODO: sql("SELECT * FROM casesensitivitytest a JOIN casesensitivitytest b ON a.a = b.a")
+ ignore("case insensitivity with scala reflection joins") {
--- End diff --
Why this test case is ignored?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/2382#discussion_r17531198
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala ---
@@ -98,12 +98,12 @@ case class Star(
override def withNullability(newNullability: Boolean) = this
override def withQualifiers(newQualifiers: Seq[String]) = this
- def expand(input: Seq[Attribute]): Seq[NamedExpression] = {
+ def expand(input: Seq[Attribute], resolver: Resolver): Seq[NamedExpression] = {
val expandedAttributes: Seq[Attribute] = table match {
// If there is no table specified, use all input attributes.
case None => input
// If there is a table, pick out attributes that are part of this table.
- case Some(t) => input.filter(_.qualifiers contains t)
+ case Some(t) => input.filter(_.qualifiers.filter(resolver(_,t)).nonEmpty)
--- End diff --
Nit: space after `,`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2382#issuecomment-55840530
Oh, one more thing, please help rename this test case: https://github.com/apache/spark/blob/0a7091e689a4c8b1e7b61e9f0873e6557f40d952/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala#L525
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2382#issuecomment-55503990
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/88/consoleFull) for PR 2382 at commit [`c2f2ec8`](https://github.com/apache/spark/commit/c2f2ec8b5967b3e69efee4ba054afb96320ac673).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2382#issuecomment-56119800
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20559/consoleFull) for PR 2382 at commit [`c21171e`](https://github.com/apache/spark/commit/c21171ecd67835dab9441ed48e0f6b76bbbf52ad).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `abstract class LogicalPlan extends QueryPlan[LogicalPlan] with Logging `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/2382#discussion_r17531262
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala ---
@@ -105,7 +119,9 @@ abstract class LogicalPlan extends QueryPlan[LogicalPlan] {
// One match, but we also need to extract the requested nested field.
case Seq((a, nestedFields)) =>
Some(Alias(nestedFields.foldLeft(a: Expression)(GetField), nestedFields.last)())
- case Seq() => None // No matches.
+ case Seq() =>
+ println(s"Could not find $name in ${input.mkString(", ")}")
--- End diff --
Use `logTrace` instead? As we did in `Analyzer`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2382#issuecomment-55504315
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20276/consoleFull) for PR 2382 at commit [`219805a`](https://github.com/apache/spark/commit/219805a2003f1f56cbf3cde37697a08ac444c124).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3414][SQL] Replace LowerCaseSchema with...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2382#discussion_r17579912
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveResolutionSuite.scala ---
@@ -57,13 +57,14 @@ class HiveResolutionSuite extends HiveComparisonTest {
.registerTempTable("caseSensitivityTest")
sql("SELECT a, b, A, B, n.a, n.b, n.A, n.B FROM caseSensitivityTest")
+ }
- println(sql("SELECT * FROM casesensitivitytest one JOIN casesensitivitytest two ON one.a = two.a").queryExecution)
-
- sql("SELECT * FROM casesensitivitytest one JOIN casesensitivitytest two ON one.a = two.a").collect()
-
- // TODO: sql("SELECT * FROM casesensitivitytest a JOIN casesensitivitytest b ON a.a = b.a")
+ ignore("case insensitivity with scala reflection joins") {
--- End diff --
Ah I see, `a.a` and `b.a` cause ambiguous attribute references here...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org