You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by hvanhovell <gi...@git.apache.org> on 2016/04/20 23:17:17 UTC
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
GitHub user hvanhovell opened a pull request:
https://github.com/apache/spark/pull/12537
[SPARK-14762][SQL] TPCDS Q90 fails to parse
### What changes were proposed in this pull request?
TPCDS Q90 fails to parse because it uses a reserved keyword as an Identifier; `AT` was used as an alias for one of the subqueries. `AT` is not a reserved keyword and should have been registerd as a in the `nonReserved` rule.
In order to prevent this from happening again I have added tests for all keywords that are non-reserved in Hive. See the `nonReserved`, `sql11ReservedKeywordsUsedAsCastFunctionName` & `sql11ReservedKeywordsUsedAsIdentifier` rules in https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g.
### How was this patch tested?
Added tests to for all Hive non reserved keywords to `TableIdentifierParserSuite`.
cc @davies
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hvanhovell/spark SPARK-14762
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12537.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12537
----
commit 5cbd4b621adcd21fe7af8133fd20f1dd3854c6c9
Author: Herman van Hovell <hv...@questtec.nl>
Date: 2016-04-20T21:04:47Z
Add most hive non reserved keywords to the non reserved list and add tests.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212610596
**[Test build #56412 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56412/consoleFull)** for PR 12537 at commit [`5cbd4b6`](https://github.com/apache/spark/commit/5cbd4b621adcd21fe7af8133fd20f1dd3854c6c9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212608725
This might fail Hive tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212773230
**[Test build #56486 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56486/consoleFull)** for PR 12537 at commit [`c9875e6`](https://github.com/apache/spark/commit/c9875e68c1fc521f4f363289bd9f7e8dd4352311).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12537#discussion_r60767536
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala ---
@@ -30,13 +64,22 @@ class TableIdentifierParserSuite extends SparkFunSuite {
// Illegal names.
intercept[ParseException](parseTableIdentifier(""))
intercept[ParseException](parseTableIdentifier("d.q.g"))
+ }
+ test("table identifier - keywords") {
// SQL Keywords.
- val keywords = Seq("select", "from", "where", "left", "right")
+ val keywords = Seq("select", "from", "where") ++ hiveNonReservedRegression
keywords.foreach { keyword =>
intercept[ParseException](parseTableIdentifier(keyword))
assert(TableIdentifier(keyword) === parseTableIdentifier(s"`$keyword`"))
assert(TableIdentifier(keyword, Option("db")) === parseTableIdentifier(s"db.`$keyword`"))
}
}
+
+ test("table identifier - non reserved keywords") {
+ // Hive keywords are allowed.
+ hiveNonReservedKeyword.foreach { nonReserved =>
+ assert(TableIdentifier(nonReserved) === parseTableIdentifier(nonReserved))
+ }
+ }
--- End diff --
Metastore allow it. That is why I agreed we can keep it without any change, but we need to add a few test cases to check if this assumption still works if we upgrade the version later. Let me submit a PR for adding these test cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212756950
**[Test build #56486 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56486/consoleFull)** for PR 12537 at commit [`c9875e6`](https://github.com/apache/spark/commit/c9875e68c1fc521f4f363289bd9f7e8dd4352311).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-213527435
**[Test build #2857 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2857/consoleFull)** for PR 12537 at commit [`d48b19e`](https://github.com/apache/spark/commit/d48b19ee75215a4c4c9b6dc2227e65e38745a2c0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212818354
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212613218
**[Test build #56413 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56413/consoleFull)** for PR 12537 at commit [`de9d8be`](https://github.com/apache/spark/commit/de9d8be933fc47681fc8c5e552798158fe197e84).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212640937
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212640744
**[Test build #56413 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56413/consoleFull)** for PR 12537 at commit [`de9d8be`](https://github.com/apache/spark/commit/de9d8be933fc47681fc8c5e552798158fe197e84).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/12537#discussion_r60547963
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala ---
@@ -30,13 +64,22 @@ class TableIdentifierParserSuite extends SparkFunSuite {
// Illegal names.
intercept[ParseException](parseTableIdentifier(""))
intercept[ParseException](parseTableIdentifier("d.q.g"))
+ }
+ test("table identifier - keywords") {
// SQL Keywords.
- val keywords = Seq("select", "from", "where", "left", "right")
+ val keywords = Seq("select", "from", "where") ++ hiveNonReservedRegression
keywords.foreach { keyword =>
intercept[ParseException](parseTableIdentifier(keyword))
assert(TableIdentifier(keyword) === parseTableIdentifier(s"`$keyword`"))
assert(TableIdentifier(keyword, Option("db")) === parseTableIdentifier(s"db.`$keyword`"))
}
}
+
+ test("table identifier - non reserved keywords") {
+ // Hive keywords are allowed.
+ hiveNonReservedKeyword.foreach { nonReserved =>
+ assert(TableIdentifier(nonReserved) === parseTableIdentifier(nonReserved))
+ }
+ }
--- End diff --
Yeah, we need to fix this. The `HiveSqlParser` passes on queries it can't parse to Hive. I think we should disable this behavior after we have integrated the `SparkSqlParser` & `HiveSqlParser`. Could you create a JIRA ticket under https://issues.apache.org/jira/browse/SPARK-14776?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212773525
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56486/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212611321
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56412/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212779775
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212773519
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12537#discussion_r60567595
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala ---
@@ -30,13 +64,22 @@ class TableIdentifierParserSuite extends SparkFunSuite {
// Illegal names.
intercept[ParseException](parseTableIdentifier(""))
intercept[ParseException](parseTableIdentifier("d.q.g"))
+ }
+ test("table identifier - keywords") {
// SQL Keywords.
- val keywords = Seq("select", "from", "where", "left", "right")
+ val keywords = Seq("select", "from", "where") ++ hiveNonReservedRegression
keywords.foreach { keyword =>
intercept[ParseException](parseTableIdentifier(keyword))
assert(TableIdentifier(keyword) === parseTableIdentifier(s"`$keyword`"))
assert(TableIdentifier(keyword, Option("db")) === parseTableIdentifier(s"db.`$keyword`"))
}
}
+
+ test("table identifier - non reserved keywords") {
+ // Hive keywords are allowed.
+ hiveNonReservedKeyword.foreach { nonReserved =>
+ assert(TableIdentifier(nonReserved) === parseTableIdentifier(nonReserved))
+ }
+ }
--- End diff --
https://issues.apache.org/jira/browse/SPARK-14802. Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212640940
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56413/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/12537#discussion_r60765517
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala ---
@@ -30,13 +64,22 @@ class TableIdentifierParserSuite extends SparkFunSuite {
// Illegal names.
intercept[ParseException](parseTableIdentifier(""))
intercept[ParseException](parseTableIdentifier("d.q.g"))
+ }
+ test("table identifier - keywords") {
// SQL Keywords.
- val keywords = Seq("select", "from", "where", "left", "right")
+ val keywords = Seq("select", "from", "where") ++ hiveNonReservedRegression
keywords.foreach { keyword =>
intercept[ParseException](parseTableIdentifier(keyword))
assert(TableIdentifier(keyword) === parseTableIdentifier(s"`$keyword`"))
assert(TableIdentifier(keyword, Option("db")) === parseTableIdentifier(s"db.`$keyword`"))
}
}
+
+ test("table identifier - non reserved keywords") {
+ // Hive keywords are allowed.
+ hiveNonReservedKeyword.foreach { nonReserved =>
+ assert(TableIdentifier(nonReserved) === parseTableIdentifier(nonReserved))
+ }
+ }
--- End diff --
@gatorsmile I think the metastore will allow names starting with underscores, since Hive allows names starting with underscores if you put the name in backticks. However we are talking about Hive, and it has surprised me before. Could you open a PR to test if the metastore allows this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-213497704
**[Test build #2857 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2857/consoleFull)** for PR 12537 at commit [`d48b19e`](https://github.com/apache/spark/commit/d48b19ee75215a4c4c9b6dc2227e65e38745a2c0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212948625
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56536/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212781058
**[Test build #56501 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56501/consoleFull)** for PR 12537 at commit [`c9875e6`](https://github.com/apache/spark/commit/c9875e68c1fc521f4f363289bd9f7e8dd4352311).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212818360
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56501/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12537
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12537#discussion_r60791101
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala ---
@@ -30,13 +64,22 @@ class TableIdentifierParserSuite extends SparkFunSuite {
// Illegal names.
intercept[ParseException](parseTableIdentifier(""))
intercept[ParseException](parseTableIdentifier("d.q.g"))
+ }
+ test("table identifier - keywords") {
// SQL Keywords.
- val keywords = Seq("select", "from", "where", "left", "right")
+ val keywords = Seq("select", "from", "where") ++ hiveNonReservedRegression
keywords.foreach { keyword =>
intercept[ParseException](parseTableIdentifier(keyword))
assert(TableIdentifier(keyword) === parseTableIdentifier(s"`$keyword`"))
assert(TableIdentifier(keyword, Option("db")) === parseTableIdentifier(s"db.`$keyword`"))
}
}
+
+ test("table identifier - non reserved keywords") {
+ // Hive keywords are allowed.
+ hiveNonReservedKeyword.foreach { nonReserved =>
+ assert(TableIdentifier(nonReserved) === parseTableIdentifier(nonReserved))
+ }
+ }
--- End diff --
In the current Parser, we still have another issue. For example, our Parser can accept
```
CREATE TABLE `tab:1` ...
```
However, Hive metastore does not allow it. We hit an exception from the metastore:
```
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: [tab:1]: is not a valid table name;
org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: [tab:1]: is not a valid table name;
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:71)
at org.apache.spark.sql.hive.HiveExternalCatalog.createTable(HiveExternalCatalog.scala:147)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createTable(SessionCatalog.scala:188)
```
In the next PR, I will add the extra logics to verify the names before calling Hive metastore APIs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212611311
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12537#discussion_r60682645
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala ---
@@ -30,13 +64,22 @@ class TableIdentifierParserSuite extends SparkFunSuite {
// Illegal names.
intercept[ParseException](parseTableIdentifier(""))
intercept[ParseException](parseTableIdentifier("d.q.g"))
+ }
+ test("table identifier - keywords") {
// SQL Keywords.
- val keywords = Seq("select", "from", "where", "left", "right")
+ val keywords = Seq("select", "from", "where") ++ hiveNonReservedRegression
keywords.foreach { keyword =>
intercept[ParseException](parseTableIdentifier(keyword))
assert(TableIdentifier(keyword) === parseTableIdentifier(s"`$keyword`"))
assert(TableIdentifier(keyword, Option("db")) === parseTableIdentifier(s"db.`$keyword`"))
}
}
+
+ test("table identifier - non reserved keywords") {
+ // Hive keywords are allowed.
+ hiveNonReservedKeyword.foreach { nonReserved =>
+ assert(TableIdentifier(nonReserved) === parseTableIdentifier(nonReserved))
+ }
+ }
--- End diff --
Another bug I found is about the table identifier. Hive does not allow `_a`. Do you want me to fix it? Or you want to do it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-213043117
**[Test build #2845 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2845/consoleFull)** for PR 12537 at commit [`d48b19e`](https://github.com/apache/spark/commit/d48b19ee75215a4c4c9b6dc2227e65e38745a2c0).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12537#discussion_r60577678
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala ---
@@ -30,13 +64,22 @@ class TableIdentifierParserSuite extends SparkFunSuite {
// Illegal names.
intercept[ParseException](parseTableIdentifier(""))
intercept[ParseException](parseTableIdentifier("d.q.g"))
+ }
+ test("table identifier - keywords") {
// SQL Keywords.
- val keywords = Seq("select", "from", "where", "left", "right")
+ val keywords = Seq("select", "from", "where") ++ hiveNonReservedRegression
keywords.foreach { keyword =>
intercept[ParseException](parseTableIdentifier(keyword))
assert(TableIdentifier(keyword) === parseTableIdentifier(s"`$keyword`"))
assert(TableIdentifier(keyword, Option("db")) === parseTableIdentifier(s"db.`$keyword`"))
}
}
+
+ test("table identifier - non reserved keywords") {
+ // Hive keywords are allowed.
+ hiveNonReservedKeyword.foreach { nonReserved =>
+ assert(TableIdentifier(nonReserved) === parseTableIdentifier(nonReserved))
+ }
+ }
--- End diff --
I am just thinking if we can capture it during the Parser? The idea is to issue a better error message when users input an incorrect error message.
How about relaxing the `.g4` parsing rule for `tableIdentifier`? That is, allowing `Parser` to accept the illegal inputs? Then, verify its correctness in the `visitTableIdentifier`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12537#discussion_r60529936
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala ---
@@ -30,13 +64,22 @@ class TableIdentifierParserSuite extends SparkFunSuite {
// Illegal names.
intercept[ParseException](parseTableIdentifier(""))
intercept[ParseException](parseTableIdentifier("d.q.g"))
+ }
+ test("table identifier - keywords") {
// SQL Keywords.
- val keywords = Seq("select", "from", "where", "left", "right")
+ val keywords = Seq("select", "from", "where") ++ hiveNonReservedRegression
keywords.foreach { keyword =>
intercept[ParseException](parseTableIdentifier(keyword))
assert(TableIdentifier(keyword) === parseTableIdentifier(s"`$keyword`"))
assert(TableIdentifier(keyword, Option("db")) === parseTableIdentifier(s"db.`$keyword`"))
}
}
+
+ test("table identifier - non reserved keywords") {
+ // Hive keywords are allowed.
+ hiveNonReservedKeyword.foreach { nonReserved =>
+ assert(TableIdentifier(nonReserved) === parseTableIdentifier(nonReserved))
+ }
+ }
--- End diff --
We need one more test case. Table identifier has illegal character `tab:1`. Then, it will directly fall into the group `hiveNativeCommands`. Then, we got a weird Hive parser error, just like
```
NoViableAltException(302@[192:1: tableName : (db= identifier DOT tab= identifier -> ^( TOK_TABNAME $db $tab) |tab= identifier -> ^( TOK_TABNAME $tab) );])
at org.antlr.runtime.DFA.noViableAlt(DFA.java:158)
at org.antlr.runtime.DFA.predict(DFA.java:116)
at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.tableName(HiveParser_FromClauseParser.java:4747)
at org.apache.hadoop.hive.ql.parse.HiveParser.tableName(HiveParser.java:45920)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-213015363
**[Test build #2845 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2845/consoleFull)** for PR 12537 at commit [`d48b19e`](https://github.com/apache/spark/commit/d48b19ee75215a4c4c9b6dc2227e65e38745a2c0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/12537#discussion_r60756114
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala ---
@@ -30,13 +64,22 @@ class TableIdentifierParserSuite extends SparkFunSuite {
// Illegal names.
intercept[ParseException](parseTableIdentifier(""))
intercept[ParseException](parseTableIdentifier("d.q.g"))
+ }
+ test("table identifier - keywords") {
// SQL Keywords.
- val keywords = Seq("select", "from", "where", "left", "right")
+ val keywords = Seq("select", "from", "where") ++ hiveNonReservedRegression
keywords.foreach { keyword =>
intercept[ParseException](parseTableIdentifier(keyword))
assert(TableIdentifier(keyword) === parseTableIdentifier(s"`$keyword`"))
assert(TableIdentifier(keyword, Option("db")) === parseTableIdentifier(s"db.`$keyword`"))
}
}
+
+ test("table identifier - non reserved keywords") {
+ // Hive keywords are allowed.
+ hiveNonReservedKeyword.foreach { nonReserved =>
+ assert(TableIdentifier(nonReserved) === parseTableIdentifier(nonReserved))
+ }
+ }
--- End diff --
@gatorsmile thanks for filing the PR.
The current identifier parsing is probably as lenient as it gets. We really cannot add much more keywords without breaking the parsing process, for instance when `LEFT` is made nonReserved the parser will turn ```select * from a left join b on a.id = b.id``` into an `INNER` join because it interprets `LEFT` as the alias for table `a`. Union/Intersect/Except are also examples of this. I do like the idea of better error messages and I don't know if ANTLR has facilities for this. Lets do this in a different PR/JIRA if we want to investigate this.
We want to support names that start with an underscore. This is mainly because of a Datasets of tuples; column names will all start with an underscore. Hive disallows underscores because `CharSetName`s start with an underscore.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-213543316
LGTM, merging into master, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212817774
**[Test build #56501 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56501/consoleFull)** for PR 12537 at commit [`c9875e6`](https://github.com/apache/spark/commit/c9875e68c1fc521f4f363289bd9f7e8dd4352311).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212948622
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212611304
**[Test build #56412 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56412/consoleFull)** for PR 12537 at commit [`5cbd4b6`](https://github.com/apache/spark/commit/5cbd4b621adcd21fe7af8133fd20f1dd3854c6c9).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212948178
**[Test build #56536 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56536/consoleFull)** for PR 12537 at commit [`d48b19e`](https://github.com/apache/spark/commit/d48b19ee75215a4c4c9b6dc2227e65e38745a2c0).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12537#discussion_r60759903
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala ---
@@ -30,13 +64,22 @@ class TableIdentifierParserSuite extends SparkFunSuite {
// Illegal names.
intercept[ParseException](parseTableIdentifier(""))
intercept[ParseException](parseTableIdentifier("d.q.g"))
+ }
+ test("table identifier - keywords") {
// SQL Keywords.
- val keywords = Seq("select", "from", "where", "left", "right")
+ val keywords = Seq("select", "from", "where") ++ hiveNonReservedRegression
keywords.foreach { keyword =>
intercept[ParseException](parseTableIdentifier(keyword))
assert(TableIdentifier(keyword) === parseTableIdentifier(s"`$keyword`"))
assert(TableIdentifier(keyword, Option("db")) === parseTableIdentifier(s"db.`$keyword`"))
}
}
+
+ test("table identifier - non reserved keywords") {
+ // Hive keywords are allowed.
+ hiveNonReservedKeyword.foreach { nonReserved =>
+ assert(TableIdentifier(nonReserved) === parseTableIdentifier(nonReserved))
+ }
+ }
--- End diff --
Thank you! @hvanhovell I see your points. Fortunately, Hive metastore does not enforce it. Otherwise, we are still unable to have names starting with underscores.
Since we are unable to trust Hive metastore APIs to issue exceptions if we inputting the illegal inputs, I am trying to see if anything we did is not consistent with Hive. Otherwise, it could cause a disaster result. For example, one of them is https://github.com/apache/spark/pull/12446. We could drop all the partitions if users input a wrong partition column name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12537#discussion_r60769766
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/TableIdentifierParserSuite.scala ---
@@ -30,13 +64,22 @@ class TableIdentifierParserSuite extends SparkFunSuite {
// Illegal names.
intercept[ParseException](parseTableIdentifier(""))
intercept[ParseException](parseTableIdentifier("d.q.g"))
+ }
+ test("table identifier - keywords") {
// SQL Keywords.
- val keywords = Seq("select", "from", "where", "left", "right")
+ val keywords = Seq("select", "from", "where") ++ hiveNonReservedRegression
keywords.foreach { keyword =>
intercept[ParseException](parseTableIdentifier(keyword))
assert(TableIdentifier(keyword) === parseTableIdentifier(s"`$keyword`"))
assert(TableIdentifier(keyword, Option("db")) === parseTableIdentifier(s"db.`$keyword`"))
}
}
+
+ test("table identifier - non reserved keywords") {
+ // Hive keywords are allowed.
+ hiveNonReservedKeyword.foreach { nonReserved =>
+ assert(TableIdentifier(nonReserved) === parseTableIdentifier(nonReserved))
+ }
+ }
--- End diff --
Will do it after https://github.com/apache/spark/pull/12081 is merged. : ) Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14762][SQL] TPCDS Q90 fails to parse
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12537#issuecomment-212915496
**[Test build #56536 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56536/consoleFull)** for PR 12537 at commit [`d48b19e`](https://github.com/apache/spark/commit/d48b19ee75215a4c4c9b6dc2227e65e38745a2c0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org