You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/05 07:33:41 UTC
[GitHub] [spark] maropu opened a new pull request #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
maropu opened a new pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
In the master, we currently don't support any nested generators, but I think supporting limited nested cases is somewhat useful for users, e.g., explode(explode(v)). This PR intends to add some logics in `ExtractGenerator` for supporting the nested generators as follows;
```
// before this PR
scala> sql("select explode(explode(array(array(1, 2), array(3))))").show()
org.apache.spark.sql.AnalysisException: Generators are not supported when it's nested in expressions, but got: explode(explode(array(array(1, 2), array(3))));
// after this PR
scala> sql("select explode(explode(array(array(1, 2), array(3))))").show()
+---+
|col|
+---+
| 1|
| 2|
| 3|
+---+
```
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
For usability.
### Does this PR introduce any user-facing change?
<!--
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Added tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595498130
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600458376
**[Test build #119966 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119966/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598154007
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-605711219
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-603820879
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25063/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597441992
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119638/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#discussion_r390676744
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2286,24 +2304,66 @@ class Analyzer(
// Holds the resolved generator, if one exists in the project list.
var resolvedGenerator: Generate = null
+ def createGenerate(g: Generator, outer: Boolean, names: Seq[String], child: LogicalPlan) = {
+ Generate(
+ g,
+ unrequiredChildIndex = Nil,
+ outer = outer,
+ qualifier = None,
+ generatorOutput = ResolveGenerate.makeGeneratorOutput(g, names),
+ child)
+ }
+
val newProjectList = projectList
.map(CleanupAliases.trimNonTopLevelAliases(_).asInstanceOf[NamedExpression])
.flatMap {
- case AliasedGenerator(generator, names, outer) if generator.childrenResolved =>
+ case AliasedGenerator(generator, names, outer) =>
// It's a sanity check, this should not happen as the previous case will throw
// exception earlier.
assert(resolvedGenerator == null, "More than one generator found in SELECT.")
- resolvedGenerator =
- Generate(
- generator,
- unrequiredChildIndex = Nil,
- outer = outer,
- qualifier = None,
- generatorOutput = ResolveGenerate.makeGeneratorOutput(generator, names),
- child)
+ if (hasInnerGenerator(generator)) {
+ // The case of multiple nested inner generators
+ def collectAdjacentGenerators(children: Seq[Expression])
Review comment:
This one too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600581592
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24709/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600738917
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598154007
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600425882
**[Test build #119966 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119966/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597441984
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598037746
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-612708265
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598027723
**[Test build #119698 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119698/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#discussion_r391337512
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2286,24 +2304,66 @@ class Analyzer(
// Holds the resolved generator, if one exists in the project list.
var resolvedGenerator: Generate = null
+ def createGenerate(g: Generator, outer: Boolean, names: Seq[String], child: LogicalPlan) = {
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595207512
**[Test build #119387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119387/testReport)** for PR 27803 at commit [`a98942a`](https://github.com/apache/spark/commit/a98942ac1b42b58b2dc899e18f2062bbe46ffa47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-604214099
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120381/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-604213522
**[Test build #120381 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120381/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598037691
**[Test build #119698 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119698/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608402321
**[Test build #120759 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120759/testReport)** for PR 27803 at commit [`ce4e9a2`](https://github.com/apache/spark/commit/ce4e9a2a437ac085910e529887a35723a733ba10).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598038777
**[Test build #119701 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119701/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600424666
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597992796
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595073051
**[Test build #119369 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119369/testReport)** for PR 27803 at commit [`a98942a`](https://github.com/apache/spark/commit/a98942ac1b42b58b2dc899e18f2062bbe46ffa47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600458631
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600738931
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119988/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598154018
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119701/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598004675
Ur, nvm. Probably, we will use this class again by the revert commit: https://github.com/apache/spark/pull/27821/files#diff-7a46f10c3cedbf013cf255564d9483cdR2346
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598154018
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119701/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597992796
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#discussion_r399906896
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/GeneratorFunctionSuite.scala
##########
@@ -344,12 +345,55 @@ class GeneratorFunctionSuite extends QueryTest with SharedSparkSession {
}
}
+ test("Supported nested inner generators") {
+ // Project cases
+ checkAnswer(
+ sql("SELECT explode(explode(array(array(1, 2), array(3))))"),
+ Row(1) :: Row(2) :: Row(3) :: Nil)
+ checkAnswer(
+ sql("SELECT array(array(4), array(5)) v").select(explode_outer(explode_outer($"v"))),
+ Row(4) :: Row(5) :: Nil)
+ checkAnswer(
+ sql("SELECT posexplode(explode(array(array(1, 2), array(3))))"),
+ Row(0, 1) :: Row(1, 2) :: Row(0, 3) :: Nil)
+ checkAnswer(
+ sql("SELECT array(array(4), array(5)) v").select(posexplode_outer(explode($"v"))),
+ Row(0, 4) :: Row(0, 5) :: Nil)
+ checkAnswer(
+ sql("SELECT inline(explode(array(array(struct(1, 'a'), struct(2, 'b')))))"),
+ Row(1, "a") :: Row(2, "b") :: Nil)
+
+ // Aggregate cases
+ checkAnswer(
+ sql("SELECT explode_outer(explode(array(array(min(v), max(v))))) FROM VALUES 1, 2, 3 t(v)"),
+ Row(1) :: Row(3) :: Nil)
+ checkAnswer(
+ sql("SELECT array(array(min(v), max(v))) ar FROM VALUES 7, 9 t(v)")
+ .select(explode(explode_outer($"ar"))),
+ Row(7) :: Row(9) :: Nil)
Review comment:
Can we add more deep test case like the following?
```
scala> sql("select explode(explode(explode(explode(array(array(array(array(1, 2), array(3, 4))))))))").show()
+---+
|col|
+---+
| 1|
| 2|
| 3|
| 4|
+---+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598037750
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119698/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597992801
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24423/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600738917
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600580180
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597375571
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24369/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-603829337
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120352/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-612708269
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25844/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595073561
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24106/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595497795
**[Test build #119424 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119424/testReport)** for PR 27803 at commit [`0ed5509`](https://github.com/apache/spark/commit/0ed5509b286521d05234a672a8634c77f3150557).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#discussion_r390676615
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2286,24 +2304,66 @@ class Analyzer(
// Holds the resolved generator, if one exists in the project list.
var resolvedGenerator: Generate = null
+ def createGenerate(g: Generator, outer: Boolean, names: Seq[String], child: LogicalPlan) = {
Review comment:
Shall we put this outside like `private def trimAlias` because this block seems to grow too big?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598039244
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-604129196
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25090/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#discussion_r391337659
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2206,17 +2220,21 @@ class Analyzer(
* @return (the [[Generator]], seq of output names, outer flag)
*/
def unapply(e: Expression): Option[(Generator, Seq[String], Boolean)] = e match {
- case Alias(GeneratorOuter(g: Generator), name) if g.resolved => Some((g, name :: Nil, true))
- case MultiAlias(GeneratorOuter(g: Generator), names) if g.resolved => Some((g, names, true))
- case Alias(g: Generator, name) if g.resolved => Some((g, name :: Nil, false))
- case MultiAlias(g: Generator, names) if g.resolved => Some((g, names, false))
+ case Alias(GeneratorOuter(g: Generator), name) => Some((g, name :: Nil, true))
+ case MultiAlias(GeneratorOuter(g: Generator), names) => Some((g, names, true))
+ case Alias(g: Generator, name) => Some((g, name :: Nil, false))
+ case MultiAlias(g: Generator, names) => Some((g, names, false))
case _ => None
}
}
def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUp {
- case Project(projectList, _) if projectList.exists(hasNestedGenerator) =>
- val nestedGenerator = projectList.find(hasNestedGenerator).get
+ // We need to resolve all the functions that might be replaced with generators
+ // before validating a plan in this rule.
+ case p if p.expressions.exists(_.find(_.isInstanceOf[UnresolvedFunction]).isDefined) => p
+
+ case Project(projectList, _) if projectList.exists(hasUnsupportedNestedGenerator) =>
+ val nestedGenerator = projectList.find(hasUnsupportedNestedGenerator).get
throw new AnalysisException("Generators are not supported when it's nested in " +
Review comment:
Sure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595599215
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119424/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597373803
Got it. Thanks for the context. Since we will not add `unnest`, this makes more sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598028085
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-605710597
Retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595498137
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24161/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600458631
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595599215
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119424/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-603820871
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-596365067
Thanks for your question, @dongjoon-hyun. Yea, actually no. But, I saw some usecases on the web where users expand nested arrays by a `unnest` function in potgresql and google big query;
```
postgres=# select unnest(ARRAY[ARRAY[1,2],ARRAY[3,4]]);
unnest
--------
1
2
3
4
(4 rows)
```
Based on this, IMO extending the existing `explode` for nested arrays seems helpful for users.
FYI: we already have the [jira](https://issues.apache.org/jira/browse/SPARK-28382) for implementing the unnest function, but we've closed the jira because of our policy as you know.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597375571
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24369/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595302785
**[Test build #119387 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119387/testReport)** for PR 27803 at commit [`a98942a`](https://github.com/apache/spark/commit/a98942ac1b42b58b2dc899e18f2062bbe46ffa47).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608267843
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-612708002
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600458637
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119966/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-603819795
**[Test build #120352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120352/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598037891
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119694/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608276010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25458/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-612709628
**[Test build #121159 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121159/testReport)** for PR 27803 at commit [`ce4e9a2`](https://github.com/apache/spark/commit/ce4e9a2a437ac085910e529887a35723a733ba10).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598038777
**[Test build #119701 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119701/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608267513
**[Test build #120747 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120747/testReport)** for PR 27803 at commit [`ce4e9a2`](https://github.com/apache/spark/commit/ce4e9a2a437ac085910e529887a35723a733ba10).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#discussion_r390677763
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2206,17 +2220,21 @@ class Analyzer(
* @return (the [[Generator]], seq of output names, outer flag)
*/
def unapply(e: Expression): Option[(Generator, Seq[String], Boolean)] = e match {
- case Alias(GeneratorOuter(g: Generator), name) if g.resolved => Some((g, name :: Nil, true))
- case MultiAlias(GeneratorOuter(g: Generator), names) if g.resolved => Some((g, names, true))
- case Alias(g: Generator, name) if g.resolved => Some((g, name :: Nil, false))
- case MultiAlias(g: Generator, names) if g.resolved => Some((g, names, false))
+ case Alias(GeneratorOuter(g: Generator), name) => Some((g, name :: Nil, true))
+ case MultiAlias(GeneratorOuter(g: Generator), names) => Some((g, names, true))
+ case Alias(g: Generator, name) => Some((g, name :: Nil, false))
+ case MultiAlias(g: Generator, names) => Some((g, names, false))
case _ => None
}
}
def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUp {
- case Project(projectList, _) if projectList.exists(hasNestedGenerator) =>
- val nestedGenerator = projectList.find(hasNestedGenerator).get
+ // We need to resolve all the functions that might be replaced with generators
+ // before validating a plan in this rule.
+ case p if p.expressions.exists(_.find(_.isInstanceOf[UnresolvedFunction]).isDefined) => p
+
+ case Project(projectList, _) if projectList.exists(hasUnsupportedNestedGenerator) =>
+ val nestedGenerator = projectList.find(hasUnsupportedNestedGenerator).get
throw new AnalysisException("Generators are not supported when it's nested in " +
Review comment:
Do we need to revise this message, `Generators are not supported when it's nested`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595599213
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-603829337
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120352/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608403130
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-604128734
**[Test build #120381 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120381/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595303402
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119387/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-612764816
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597373835
Retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598027723
**[Test build #119698 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119698/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598037891
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119694/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598039257
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24430/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597940638
Thanks for the review, @dongjoon-hyun. I'll update soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595488872
Thanks for the re-run, @HyukjinKwon . I'm checking the test failures.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595498130
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-603829274
**[Test build #120352 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120352/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
* This patch **fails to generate documentation**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-603819392
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597994188
**[Test build #119694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119694/testReport)** for PR 27803 at commit [`0b3a3de`](https://github.com/apache/spark/commit/0b3a3de1011e1b1f5706b0beead4b6f98d09f283).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-604129184
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595205375
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600426225
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24690/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598037885
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-596281185
Hi, @maropu . Just a question, is there any reference to support this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-603820871
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608202859
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598038414
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600737501
**[Test build #119988 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119988/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595084202
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598037885
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598028085
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-612764825
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121159/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600738931
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119988/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595208049
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595084209
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119369/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608202599
**[Test build #120747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120747/testReport)** for PR 27803 at commit [`ce4e9a2`](https://github.com/apache/spark/commit/ce4e9a2a437ac085910e529887a35723a733ba10).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600580987
**[Test build #119988 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119988/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608274108
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#discussion_r391337609
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/GeneratorFunctionSuite.scala
##########
@@ -344,12 +344,35 @@ class GeneratorFunctionSuite extends QueryTest with SharedSparkSession {
}
}
+ test("Supported nested inner generators") {
Review comment:
Yea, I'll try.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-605759929
BTW, @maropu . After rethinking about this, this PR looks okay because it works like the following. So, we support more deep invocation if we add more `explode`.
```
scala> sql("select explode(explode(explode(array(array(array(array(1, 2), array(3, 4)))))))").show()
+------+
| col|
+------+
|[1, 2]|
|[3, 4]|
+------+
scala> sql("select explode(explode(explode(explode(array(array(array(array(1, 2), array(3, 4))))))))").show()
+---+
|col|
+---+
| 1|
| 2|
| 3|
| 4|
+---+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597375193
**[Test build #119638 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119638/testReport)** for PR 27803 at commit [`0ed5509`](https://github.com/apache/spark/commit/0ed5509b286521d05234a672a8634c77f3150557).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-603819795
**[Test build #120352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120352/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-603820879
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25063/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608202859
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608276004
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-605711219
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598003381
Btw, I noticed now that `UserDefinedGenerator` is not used anywhere and we need to keep this class in the master?
```
maropu @ ~/spark-master: $ find . -type f | grep -e "\.scala$" | xargs grep UserDefinedGenerator
./sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala:case class UserDefinedGenerator(
./sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala: override def toString: String = s"UserDefinedGenerator(${children.mkString(",")})"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-604214099
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120381/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608267857
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120747/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-605758592
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600581587
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-605758180
**[Test build #120560 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120560/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600425882
**[Test build #119966 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119966/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600458637
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119966/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595599213
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598039244
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-605758601
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120560/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-605759929
BTW, @maropu . After rethinking about this, the current approach of this PR also looks okay because it works like the following. In other words, this PR supports more deep-invocations if we add more `explode`.
```
scala> sql("select explode(explode(explode(array(array(array(array(1, 2), array(3, 4)))))))").show()
+------+
| col|
+------+
|[1, 2]|
|[3, 4]|
+------+
scala> sql("select explode(explode(explode(explode(array(array(array(array(1, 2), array(3, 4))))))))").show()
+---+
|col|
+---+
| 1|
| 2|
| 3|
| 4|
+---+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608202866
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25446/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608267843
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-612764187
**[Test build #121159 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121159/testReport)** for PR 27803 at commit [`ce4e9a2`](https://github.com/apache/spark/commit/ce4e9a2a437ac085910e529887a35723a733ba10).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-612764816
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595498137
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24161/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608275515
**[Test build #120759 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120759/testReport)** for PR 27803 at commit [`ce4e9a2`](https://github.com/apache/spark/commit/ce4e9a2a437ac085910e529887a35723a733ba10).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595073548
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-604129196
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25090/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608202866
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25446/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597375193
**[Test build #119638 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119638/testReport)** for PR 27803 at commit [`0ed5509`](https://github.com/apache/spark/commit/0ed5509b286521d05234a672a8634c77f3150557).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597375565
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595208059
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24125/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598152731
**[Test build #119701 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119701/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-604214095
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-605758592
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-605711222
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25266/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-605759929
BTW, @maropu . After rethinking about this, the approach of this PR looks okay because it works like the following. So, we support more deep invocation if we add more `explode`.
```
scala> sql("select explode(explode(explode(array(array(array(array(1, 2), array(3, 4)))))))").show()
+------+
| col|
+------+
|[1, 2]|
|[3, 4]|
+------+
scala> sql("select explode(explode(explode(explode(array(array(array(array(1, 2), array(3, 4))))))))").show()
+---+
|col|
+---+
| 1|
| 2|
| 3|
| 4|
+---+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-603829329
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-605711031
**[Test build #120560 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120560/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608276004
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-605758601
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120560/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600426222
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600581592
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24709/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597441984
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598037750
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119698/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608403135
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120759/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595303389
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595303402
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119387/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595303389
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595084209
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119369/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598028090
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24427/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-603829329
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595073561
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24106/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-605711031
**[Test build #120560 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120560/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-605711222
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25266/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598028090
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24427/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597375565
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597992801
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24423/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595598771
**[Test build #119424 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119424/testReport)** for PR 27803 at commit [`0ed5509`](https://github.com/apache/spark/commit/0ed5509b286521d05234a672a8634c77f3150557).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-612709628
**[Test build #121159 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121159/testReport)** for PR 27803 at commit [`ce4e9a2`](https://github.com/apache/spark/commit/ce4e9a2a437ac085910e529887a35723a733ba10).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595497795
**[Test build #119424 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119424/testReport)** for PR 27803 at commit [`0ed5509`](https://github.com/apache/spark/commit/0ed5509b286521d05234a672a8634c77f3150557).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598039257
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24430/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595208049
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600580987
**[Test build #119988 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119988/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-604129184
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-612764825
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121159/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-605759929
BTW, @maropu . After rethinking about this, the approach of this PR looks okay because it works like the following. In other words, this PR supports more deep-invocations if we add more `explode`.
```
scala> sql("select explode(explode(explode(array(array(array(array(1, 2), array(3, 4)))))))").show()
+------+
| col|
+------+
|[1, 2]|
|[3, 4]|
+------+
scala> sql("select explode(explode(explode(explode(array(array(array(array(1, 2), array(3, 4))))))))").show()
+---+
|col|
+---+
| 1|
| 2|
| 3|
| 4|
+---+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-604127497
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-612708265
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#discussion_r391337534
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2286,24 +2304,66 @@ class Analyzer(
// Holds the resolved generator, if one exists in the project list.
var resolvedGenerator: Generate = null
+ def createGenerate(g: Generator, outer: Boolean, names: Seq[String], child: LogicalPlan) = {
+ Generate(
+ g,
+ unrequiredChildIndex = Nil,
+ outer = outer,
+ qualifier = None,
+ generatorOutput = ResolveGenerate.makeGeneratorOutput(g, names),
+ child)
+ }
+
val newProjectList = projectList
.map(CleanupAliases.trimNonTopLevelAliases(_).asInstanceOf[NamedExpression])
.flatMap {
- case AliasedGenerator(generator, names, outer) if generator.childrenResolved =>
+ case AliasedGenerator(generator, names, outer) =>
// It's a sanity check, this should not happen as the previous case will throw
// exception earlier.
assert(resolvedGenerator == null, "More than one generator found in SELECT.")
- resolvedGenerator =
- Generate(
- generator,
- unrequiredChildIndex = Nil,
- outer = outer,
- qualifier = None,
- generatorOutput = ResolveGenerate.makeGeneratorOutput(generator, names),
- child)
+ if (hasInnerGenerator(generator)) {
+ // The case of multiple nested inner generators
+ def collectAdjacentGenerators(children: Seq[Expression])
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608276010
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25458/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608403130
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595084202
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#discussion_r391337659
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2206,17 +2220,21 @@ class Analyzer(
* @return (the [[Generator]], seq of output names, outer flag)
*/
def unapply(e: Expression): Option[(Generator, Seq[String], Boolean)] = e match {
- case Alias(GeneratorOuter(g: Generator), name) if g.resolved => Some((g, name :: Nil, true))
- case MultiAlias(GeneratorOuter(g: Generator), names) if g.resolved => Some((g, names, true))
- case Alias(g: Generator, name) if g.resolved => Some((g, name :: Nil, false))
- case MultiAlias(g: Generator, names) if g.resolved => Some((g, names, false))
+ case Alias(GeneratorOuter(g: Generator), name) => Some((g, name :: Nil, true))
+ case MultiAlias(GeneratorOuter(g: Generator), names) => Some((g, names, true))
+ case Alias(g: Generator, name) => Some((g, name :: Nil, false))
+ case MultiAlias(g: Generator, names) => Some((g, names, false))
case _ => None
}
}
def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUp {
- case Project(projectList, _) if projectList.exists(hasNestedGenerator) =>
- val nestedGenerator = projectList.find(hasNestedGenerator).get
+ // We need to resolve all the functions that might be replaced with generators
+ // before validating a plan in this rule.
+ case p if p.expressions.exists(_.find(_.isInstanceOf[UnresolvedFunction]).isDefined) => p
+
+ case Project(projectList, _) if projectList.exists(hasUnsupportedNestedGenerator) =>
+ val nestedGenerator = projectList.find(hasUnsupportedNestedGenerator).get
throw new AnalysisException("Generators are not supported when it's nested in " +
Review comment:
Yea, ok, I'll update the message.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600426225
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24690/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597441449
**[Test build #119638 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119638/testReport)** for PR 27803 at commit [`0ed5509`](https://github.com/apache/spark/commit/0ed5509b286521d05234a672a8634c77f3150557).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595208059
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24125/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595073548
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595084139
**[Test build #119369 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119369/testReport)** for PR 27803 at commit [`a98942a`](https://github.com/apache/spark/commit/a98942ac1b42b58b2dc899e18f2062bbe46ffa47).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608403135
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120759/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-605719532
Sure, I'll check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608275515
**[Test build #120759 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120759/testReport)** for PR 27803 at commit [`ce4e9a2`](https://github.com/apache/spark/commit/ce4e9a2a437ac085910e529887a35723a733ba10).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#discussion_r399906896
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/GeneratorFunctionSuite.scala
##########
@@ -344,12 +345,55 @@ class GeneratorFunctionSuite extends QueryTest with SharedSparkSession {
}
}
+ test("Supported nested inner generators") {
+ // Project cases
+ checkAnswer(
+ sql("SELECT explode(explode(array(array(1, 2), array(3))))"),
+ Row(1) :: Row(2) :: Row(3) :: Nil)
+ checkAnswer(
+ sql("SELECT array(array(4), array(5)) v").select(explode_outer(explode_outer($"v"))),
+ Row(4) :: Row(5) :: Nil)
+ checkAnswer(
+ sql("SELECT posexplode(explode(array(array(1, 2), array(3))))"),
+ Row(0, 1) :: Row(1, 2) :: Row(0, 3) :: Nil)
+ checkAnswer(
+ sql("SELECT array(array(4), array(5)) v").select(posexplode_outer(explode($"v"))),
+ Row(0, 4) :: Row(0, 5) :: Nil)
+ checkAnswer(
+ sql("SELECT inline(explode(array(array(struct(1, 'a'), struct(2, 'b')))))"),
+ Row(1, "a") :: Row(2, "b") :: Nil)
+
+ // Aggregate cases
+ checkAnswer(
+ sql("SELECT explode_outer(explode(array(array(min(v), max(v))))) FROM VALUES 1, 2, 3 t(v)"),
+ Row(1) :: Row(3) :: Nil)
+ checkAnswer(
+ sql("SELECT array(array(min(v), max(v))) ar FROM VALUES 7, 9 t(v)")
+ .select(explode(explode_outer($"ar"))),
+ Row(7) :: Row(9) :: Nil)
Review comment:
Can we add more deep test case like the following?
```scala
scala> sql("select explode(explode(explode(explode(array(array(array(array(1, 2), array(3, 4))))))))").show()
+---+
|col|
+---+
| 1|
| 2|
| 3|
| 4|
+---+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-604214095
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600581587
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597994188
**[Test build #119694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119694/testReport)** for PR 27803 at commit [`0b3a3de`](https://github.com/apache/spark/commit/0b3a3de1011e1b1f5706b0beead4b6f98d09f283).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595073051
**[Test build #119369 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119369/testReport)** for PR 27803 at commit [`a98942a`](https://github.com/apache/spark/commit/a98942ac1b42b58b2dc899e18f2062bbe46ffa47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-612708269
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25844/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608267857
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120747/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-597441992
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119638/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-608202599
**[Test build #120747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120747/testReport)** for PR 27803 at commit [`ce4e9a2`](https://github.com/apache/spark/commit/ce4e9a2a437ac085910e529887a35723a733ba10).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598037746
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-600426222
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#discussion_r390677218
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/GeneratorFunctionSuite.scala
##########
@@ -344,12 +344,35 @@ class GeneratorFunctionSuite extends QueryTest with SharedSparkSession {
}
}
+ test("Supported nested inner generators") {
Review comment:
Can we add more generators? Or, shall we mention `explode/explode_outer`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27803:
[SPARK-31049][SQL] Support nested adjacent generators, e.g.,
explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-604128734
**[Test build #120381 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120381/testReport)** for PR 27803 at commit [`2bebb5b`](https://github.com/apache/spark/commit/2bebb5b3564a09d5892d9b21f3ebd6853c491ab0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-595207512
**[Test build #119387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119387/testReport)** for PR 27803 at commit [`a98942a`](https://github.com/apache/spark/commit/a98942ac1b42b58b2dc899e18f2062bbe46ffa47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL]
Support nested adjacent generators, e.g., explode(explode(v))
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
URL: https://github.com/apache/spark/pull/27803#issuecomment-598037692
**[Test build #119694 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119694/testReport)** for PR 27803 at commit [`0b3a3de`](https://github.com/apache/spark/commit/0b3a3de1011e1b1f5706b0beead4b6f98d09f283).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org