You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by maropu <gi...@git.apache.org> on 2017/05/10 00:45:37 UTC
[GitHub] spark pull request #17928: [SPARK-20311][SQL] Support aliases for table valu...
GitHub user maropu opened a pull request:
https://github.com/apache/spark/pull/17928
[SPARK-20311][SQL] Support aliases for table value functions
## What changes were proposed in this pull request?
This pr added parsing rules to support aliases in table value functions.
The previous pr (#17666) has been reverted because of the regression. This new pr fixed the regression and add tests in `SQLQueryTestSuite`.
## How was this patch tested?
Added tests in `PlanParserSuite` and `SQLQueryTestSuite`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maropu/spark SPARK-20311-3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17928.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17928
----
commit 47b923b7766c7f8cc9604287d4a88683c8c370b2
Author: Takeshi Yamamuro <ya...@apache.org>
Date: 2017-04-18T00:39:26Z
Support aliases for table value functions
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/17928
@cloud-fan ok, could you check again? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17928
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17928: [SPARK-20311][SQL] Support aliases for table valu...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17928#discussion_r115663190
--- Diff: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ---
@@ -472,15 +472,23 @@ identifierComment
;
relationPrimary
- : tableIdentifier sample? (AS? strictIdentifier)? #tableName
- | '(' queryNoWith ')' sample? (AS? strictIdentifier)? #aliasedQuery
- | '(' relation ')' sample? (AS? strictIdentifier)? #aliasedRelation
- | inlineTable #inlineTableDefault2
- | identifier '(' (expression (',' expression)*)? ')' #tableValuedFunction
+ : tableIdentifier sample? (AS? strictIdentifier)? #tableName
+ | '(' queryNoWith ')' sample? (AS? strictIdentifier)? #aliasedQuery
+ | '(' relation ')' sample? (AS? strictIdentifier)? #aliasedRelation
+ | inlineTable #inlineTableDefault2
+ | functionTable #tableValuedFunction
;
inlineTable
- : VALUES expression (',' expression)* (AS? identifier identifierList?)?
+ : VALUES expression (',' expression)* tableAlias
+ ;
+
+functionTable
+ : identifier '(' (expression (',' expression)*)? ')' tableAlias
+ ;
+
+tableAlias
+ : (AS? strictIdentifier identifierList?)?
--- End diff --
This also hits another bug in inline tables. Maybe you also can include the following query in the test case `inline-table.sql`?
```
sql("SELECT * FROM VALUES (\"one\", 1), (\"two\", 2), (\"three\", null) CROSS JOIN VALUES (\"one\", 1), (\"two\", 2), (\"three\", null)")
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17928: [SPARK-20311][SQL] Support aliases for table valu...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17928#discussion_r115663412
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala ---
@@ -468,7 +468,18 @@ class PlanParserSuite extends PlanTest {
test("table valued function") {
assertEqual(
"select * from range(2)",
- UnresolvedTableValuedFunction("range", Literal(2) :: Nil).select(star()))
+ UnresolvedTableValuedFunction("range", Literal(2) :: Nil, Seq.empty).select(star()))
+ }
+
+ test("SPARK-20311 range(N) as alias") {
+ assertEqual(
+ "select * from range(10) AS t",
--- End diff --
Nit: ` SELECT * FROM range(10) AS t`
BTW, we prefer to use upper case for the SQL keywords.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17928
**[Test build #76737 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76737/testReport)** for PR 17928 at commit [`29281b1`](https://github.com/apache/spark/commit/29281b1af00ced947173701378e9ad1b67e4924c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17928
**[Test build #76791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76791/testReport)** for PR 17928 at commit [`a7a732b`](https://github.com/apache/spark/commit/a7a732bdbef47b4b4695843ab8005a4a85b037da).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17928: [SPARK-20311][SQL] Support aliases for table valu...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17928#discussion_r115638027
--- Diff: sql/core/src/test/resources/sql-tests/results/subquery/exists-subquery/exists-cte.sql.out ---
@@ -190,11 +190,7 @@ WHERE NOT EXISTS (SELECT dept_id,
HAVING count(*) < 1)
GROUP BY emp_name
-- !query 7 schema
-struct<emp_name:string,sum(bonus_amt):double>
+struct<>
-- !query 7 output
-emp 1 30.0
-emp 2 400.0
-emp 3 300.0
-emp 4 100.0
-emp 5 1000.0
-emp 6 - no dept 500.0
+org.apache.spark.SparkException
+Exception thrown in awaitResult:
--- End diff --
Fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17928
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17928: [SPARK-20311][SQL] Support aliases for table valu...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17928#discussion_r115636065
--- Diff: sql/core/src/test/resources/sql-tests/inputs/table-valued-functions.sql ---
@@ -24,3 +24,7 @@ select * from RaNgE(2);
-- Explain
EXPLAIN select * from RaNgE(2);
+
+-- cross-join table valued functions
+set spark.sql.crossJoin.enabled=true;
+EXPLAIN EXTENDED select * from range(3) cross join range(3);
--- End diff --
we can just check the answer instead of running explain
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17928
**[Test build #76770 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76770/testReport)** for PR 17928 at commit [`54a05da`](https://github.com/apache/spark/commit/54a05daaf26c9ba7228e3d9c35085c3bc6015d65).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17928
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76784/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17928
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76712/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17928
**[Test build #76712 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76712/testReport)** for PR 17928 at commit [`47b923b`](https://github.com/apache/spark/commit/47b923b7766c7f8cc9604287d4a88683c8c370b2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class UnresolvedTableValuedFunction(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17928: [SPARK-20311][SQL] Support aliases for table valu...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17928#discussion_r115913870
--- Diff: sql/core/src/test/resources/sql-tests/inputs/inline-table.sql ---
@@ -49,3 +49,6 @@ select * from values ("one", count(1)), ("two", 2) as data(a, b);
-- string to timestamp
select * from values (timestamp('1991-12-06 00:00:00.0'), array(timestamp('1991-12-06 01:00:00.0'), timestamp('1991-12-06 12:00:00.0'))) as data(a, b);
+
+-- cross-join inline tables
+SELECT * FROM VALUES ('one', 1), ('three', null) CROSS JOIN VALUES ('one', 1), ('three', null);
--- End diff --
does this expose the bug? If we treat CROSS as an alias, we still the same result. how about we run EXPLAIN?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17928
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76716/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17928: [SPARK-20311][SQL] Support aliases for table valu...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17928#discussion_r115636644
--- Diff: sql/core/src/test/resources/sql-tests/results/subquery/exists-subquery/exists-cte.sql.out ---
@@ -190,11 +190,7 @@ WHERE NOT EXISTS (SELECT dept_id,
HAVING count(*) < 1)
GROUP BY emp_name
-- !query 7 schema
-struct<emp_name:string,sum(bonus_amt):double>
+struct<>
-- !query 7 output
-emp 1 30.0
-emp 2 400.0
-emp 3 300.0
-emp 4 100.0
-emp 5 1000.0
-emp 6 - no dept 500.0
+org.apache.spark.SparkException
+Exception thrown in awaitResult:
--- End diff --
oh, my bad. I'll check again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17928
Thanks! Merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17928: [SPARK-20311][SQL] Support aliases for table valu...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17928#discussion_r115664938
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala ---
@@ -468,7 +468,18 @@ class PlanParserSuite extends PlanTest {
test("table valued function") {
assertEqual(
"select * from range(2)",
- UnresolvedTableValuedFunction("range", Literal(2) :: Nil).select(star()))
+ UnresolvedTableValuedFunction("range", Literal(2) :: Nil, Seq.empty).select(star()))
+ }
+
+ test("SPARK-20311 range(N) as alias") {
+ assertEqual(
+ "select * from range(10) AS t",
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17928
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17928
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76770/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17928
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76791/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17928
**[Test build #76716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76716/testReport)** for PR 17928 at commit [`0904fc9`](https://github.com/apache/spark/commit/0904fc9feabf348c92ebeaff0fd2a506c1cec128).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17928: [SPARK-20311][SQL] Support aliases for table valu...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17928#discussion_r115791768
--- Diff: sql/core/src/test/resources/sql-tests/inputs/table-valued-functions.sql ---
@@ -26,5 +26,5 @@ select * from RaNgE(2);
EXPLAIN select * from RaNgE(2);
-- cross-join table valued functions
-set spark.sql.crossJoin.enabled=true;
-EXPLAIN EXTENDED select * from range(3) cross join range(3);
+SET spark.sql.crossJoin.enabled=true;
--- End diff --
Could you remove this line? If we specify `CROSS JOIN` in the query, no need to set this parm.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17928
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17928
LGTM, cc @gatorsmile to take another look
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17928: [SPARK-20311][SQL] Support aliases for table valu...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17928#discussion_r115792064
--- Diff: sql/core/src/test/resources/sql-tests/inputs/table-valued-functions.sql ---
@@ -24,3 +24,7 @@ select * from RaNgE(2);
-- Explain
EXPLAIN select * from RaNgE(2);
+
+-- cross-join table valued functions
+SET spark.sql.crossJoin.enabled=true;
--- End diff --
Could you remove this line? If we specify CROSS JOIN in the query, no need to set this parm.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17928
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17928: [SPARK-20311][SQL] Support aliases for table valu...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17928#discussion_r115636382
--- Diff: sql/core/src/test/resources/sql-tests/results/subquery/exists-subquery/exists-cte.sql.out ---
@@ -190,11 +190,7 @@ WHERE NOT EXISTS (SELECT dept_id,
HAVING count(*) < 1)
GROUP BY emp_name
-- !query 7 schema
-struct<emp_name:string,sum(bonus_amt):double>
+struct<>
-- !query 7 output
-emp 1 30.0
-emp 2 400.0
-emp 3 300.0
-emp 4 100.0
-emp 5 1000.0
-emp 6 - no dept 500.0
+org.apache.spark.SparkException
+Exception thrown in awaitResult:
--- End diff --
are we breaking existing queries?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17928: [SPARK-20311][SQL] Support aliases for table valu...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/17928
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17928
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17928
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76737/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17928
**[Test build #76770 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76770/testReport)** for PR 17928 at commit [`54a05da`](https://github.com/apache/spark/commit/54a05daaf26c9ba7228e3d9c35085c3bc6015d65).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17928
**[Test build #76737 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76737/testReport)** for PR 17928 at commit [`29281b1`](https://github.com/apache/spark/commit/29281b1af00ced947173701378e9ad1b67e4924c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17928
**[Test build #76712 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76712/testReport)** for PR 17928 at commit [`47b923b`](https://github.com/apache/spark/commit/47b923b7766c7f8cc9604287d4a88683c8c370b2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17928
**[Test build #76784 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76784/testReport)** for PR 17928 at commit [`a7a732b`](https://github.com/apache/spark/commit/a7a732bdbef47b4b4695843ab8005a4a85b037da).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17928: [SPARK-20311][SQL] Support aliases for table valu...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17928#discussion_r115663460
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala ---
@@ -468,7 +468,18 @@ class PlanParserSuite extends PlanTest {
test("table valued function") {
assertEqual(
"select * from range(2)",
- UnresolvedTableValuedFunction("range", Literal(2) :: Nil).select(star()))
+ UnresolvedTableValuedFunction("range", Literal(2) :: Nil, Seq.empty).select(star()))
+ }
+
+ test("SPARK-20311 range(N) as alias") {
+ assertEqual(
+ "select * from range(10) AS t",
--- End diff --
You also can update the similar issues in your test cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17928
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17928: [SPARK-20311][SQL] Support aliases for table valu...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17928#discussion_r115914124
--- Diff: sql/core/src/test/resources/sql-tests/inputs/inline-table.sql ---
@@ -49,3 +49,6 @@ select * from values ("one", count(1)), ("two", 2) as data(a, b);
-- string to timestamp
select * from values (timestamp('1991-12-06 00:00:00.0'), array(timestamp('1991-12-06 01:00:00.0'), timestamp('1991-12-06 12:00:00.0'))) as data(a, b);
+
+-- cross-join inline tables
+SELECT * FROM VALUES ('one', 1), ('three', null) CROSS JOIN VALUES ('one', 1), ('three', null);
--- End diff --
Aha, ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17928
**[Test build #76791 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76791/testReport)** for PR 17928 at commit [`a7a732b`](https://github.com/apache/spark/commit/a7a732bdbef47b4b4695843ab8005a4a85b037da).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17928: [SPARK-20311][SQL] Support aliases for table valu...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17928#discussion_r115672645
--- Diff: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ---
@@ -472,15 +472,23 @@ identifierComment
;
relationPrimary
- : tableIdentifier sample? (AS? strictIdentifier)? #tableName
- | '(' queryNoWith ')' sample? (AS? strictIdentifier)? #aliasedQuery
- | '(' relation ')' sample? (AS? strictIdentifier)? #aliasedRelation
- | inlineTable #inlineTableDefault2
- | identifier '(' (expression (',' expression)*)? ')' #tableValuedFunction
+ : tableIdentifier sample? (AS? strictIdentifier)? #tableName
+ | '(' queryNoWith ')' sample? (AS? strictIdentifier)? #aliasedQuery
+ | '(' relation ')' sample? (AS? strictIdentifier)? #aliasedRelation
+ | inlineTable #inlineTableDefault2
+ | functionTable #tableValuedFunction
;
inlineTable
- : VALUES expression (',' expression)* (AS? identifier identifierList?)?
+ : VALUES expression (',' expression)* tableAlias
+ ;
+
+functionTable
+ : identifier '(' (expression (',' expression)*)? ')' tableAlias
+ ;
+
+tableAlias
+ : (AS? strictIdentifier identifierList?)?
--- End diff --
Added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17928
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17928: [SPARK-20311][SQL] Support aliases for table value funct...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17928
**[Test build #76716 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76716/testReport)** for PR 17928 at commit [`0904fc9`](https://github.com/apache/spark/commit/0904fc9feabf348c92ebeaff0fd2a506c1cec128).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class UnresolvedTableValuedFunction(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17928: [SPARK-20311][SQL] Support aliases for table valu...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17928#discussion_r115895718
--- Diff: sql/core/src/test/resources/sql-tests/inputs/table-valued-functions.sql ---
@@ -24,3 +24,7 @@ select * from RaNgE(2);
-- Explain
EXPLAIN select * from RaNgE(2);
+
+-- cross-join table valued functions
+SET spark.sql.crossJoin.enabled=true;
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org