You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/23 09:45:55 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
AngersZhuuuu opened a new pull request #33810:
URL: https://github.com/apache/spark/pull/33810
#### What changes were proposed in this pull request?
This PR fixes an issue that field names of structs generated by arrays_zip function could be unexpectedly re-written by analyzer/optimizer.
Here is an example.
```
val df = sc.parallelize(Seq((Array(1, 2), Array(3, 4)))).toDF("a1", "b1").selectExpr("arrays_zip(a1, b1) as zipped")
df.printSchema
root
|-- zipped: array (nullable = true)
| |-- element: struct (containsNull = false)
| | |-- a1: integer (nullable = true) // OK. a1 is expected name
| | |-- b1: integer (nullable = true) // OK. b1 is expected name
df.explain
== Physical Plan ==
*(1) Project [arrays_zip(_1#3, _2#4) AS zipped#12] // Not OK. field names are re-written as _1 and _2 respectively
df.write.parquet("/tmp/test.parquet")
val df2 = spark.read.parquet("/tmp/test.parquet")
df2.printSchema
root
|-- zipped: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- _1: integer (nullable = true) // Not OK. a1 is expected but got _1
| | |-- _2: integer (nullable = true) // Not OK. b1 is expected but got _2
```
This issue happens when aliases are eliminated by AliasHelper.replaceAliasButKeepName or AliasHelper.trimNonTopLevelAliases called via analyzer/optimizer
spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
Line 883 in b89cd8d
upper.map(replaceAliasButKeepName(_, aliases))
spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
Line 3759 in b89cd8d
val cleanedProjectList = projectList.map(trimNonTopLevelAliases)
I investigated functions which can be affected this issue but I found only arrays_zip so far.
To fix this issue, this PR changes the definition of ArraysZip to retain field names to avoid being re-written by analyzer/optimizer.
### Why are the changes needed?
This is apparently a bug.
### Does this PR introduce any user-facing change?
No. After this change, the field names are no longer re-written but it should be expected behavior for users.
#### How was this patch tested?
New tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904539874
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47222/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903807678
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142694/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903943767
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142704/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904609297
**[Test build #142728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142728/testReport)** for PR 33810 at commit [`8933e2e`](https://github.com/apache/spark/commit/8933e2e22693b16f1e7c023bd11461d3fe17c7e7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904645241
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47228/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904493370
**[Test build #142722 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142722/testReport)** for PR 33810 at commit [`e633591`](https://github.com/apache/spark/commit/e633591c2b0d1a897c17206e7cc9138bf2cedb0b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33810:
URL: https://github.com/apache/spark/pull/33810#discussion_r694093090
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
##########
@@ -174,16 +174,35 @@ case class MapKeys(child: Expression)
""",
group = "array_funcs",
since = "2.4.0")
-case class ArraysZip(children: Seq[Expression]) extends Expression with ExpectsInputTypes {
+case class ArraysZip(children: Seq[Expression], names: Seq[Expression])
+ extends Expression with ExpectsInputTypes {
+
+ def this(children: Seq[Expression]) = {
+ this(
+ children,
+ children.zipWithIndex.map {
+ case (u: UnresolvedAttribute, _) => Literal(u.nameParts.last)
+ case (e: NamedExpression, _) if e.resolved => Literal(e.name)
+ case (e: NamedExpression, _) => NamePlaceholder
+ case (_, idx) => Literal(idx.toString)
+ })
+ }
+
+ if (children.size != names.size) {
+ throw new IllegalArgumentException(
+ "The numbers of zipped arrays and field names should be the same")
+ }
+
+ override lazy val resolved: Boolean =
+ childrenResolved && checkInputDataTypes().isSuccess && names.forall(_.resolved)
override def inputTypes: Seq[AbstractDataType] = Seq.fill(children.length)(ArrayType)
@transient override lazy val dataType: DataType = {
- val fields = children.zip(arrayElementTypes).zipWithIndex.map {
- case ((expr: NamedExpression, elementType), _) =>
- StructField(expr.name, elementType, nullable = true)
- case ((_, elementType), idx) =>
- StructField(idx.toString, elementType, nullable = true)
+ val fields = arrayElementTypes.zip(names).map {
+ case (elementType, Literal(name, StringType)) =>
+ StructField(name.toString, elementType, nullable = true)
+ case _ => StructField("", NullType)
Review comment:
> why does 3.1 need this change?
Forgot remove ==， remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903689542
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47197/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903889488
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142695/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904820254
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142728/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903939326
**[Test build #142704 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142704/testReport)** for PR 33810 at commit [`1cbc87b`](https://github.com/apache/spark/commit/1cbc87b726a9df44c27b9b261639c5240e887753).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903689542
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47197/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903807678
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142694/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904609297
**[Test build #142728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142728/testReport)** for PR 33810 at commit [`8933e2e`](https://github.com/apache/spark/commit/8933e2e22693b16f1e7c023bd11461d3fe17c7e7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904668948
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142722/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904820254
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142728/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33810:
URL: https://github.com/apache/spark/pull/33810#discussion_r694787137
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
##########
@@ -174,16 +175,36 @@ case class MapKeys(child: Expression)
""",
group = "array_funcs",
since = "2.4.0")
-case class ArraysZip(children: Seq[Expression]) extends Expression with ExpectsInputTypes {
+case class ArraysZip(children: Seq[Expression], names: Seq[Expression])
+ extends Expression with ExpectsInputTypes {
+
+ def this(children: Seq[Expression]) = {
+ this(
+ children,
+ children.zipWithIndex.map {
+ case (u: UnresolvedAttribute, _) => Literal(u.nameParts.last)
+ case (e: NamedExpression, _) if e.resolved => Literal(e.name)
+ case (e: NamedExpression, _) => NamePlaceholder
+ case (_, idx) => Literal(idx.toString)
+ })
+ }
+
+ if (children.size != names.size) {
+ throw new IllegalArgumentException(
+ "The numbers of zipped arrays and field names should be the same")
+ }
+
+ override lazy val resolved: Boolean =
+ childrenResolved && checkInputDataTypes().isSuccess && names.forall(_.resolved)
override def inputTypes: Seq[AbstractDataType] = Seq.fill(children.length)(ArrayType)
@transient override lazy val dataType: DataType = {
- val fields = children.zip(arrayElementTypes).zipWithIndex.map {
- case ((expr: NamedExpression, elementType), _) =>
- StructField(expr.name, elementType, nullable = true)
- case ((_, elementType), idx) =>
- StructField(idx.toString, elementType, nullable = true)
+ val fields = arrayElementTypes.zip(names).map {
+ case (elementType, Literal(name, StringType)) =>
+ StructField(name.toString, elementType, nullable = true)
+ case _ =>
+ throw new AnalysisException("Schema name of arrays_zip should be string literal.")
Review comment:
This means bug, I think IllegalStateException is better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903805362
**[Test build #142694 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142694/testReport)** for PR 33810 at commit [`ff1353e`](https://github.com/apache/spark/commit/ff1353e4130ffaa83aee9c1134c0c1242abea2fc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904818406
**[Test build #142728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142728/testReport)** for PR 33810 at commit [`8933e2e`](https://github.com/apache/spark/commit/8933e2e22693b16f1e7c023bd11461d3fe17c7e7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903665833
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47196/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #33810:
URL: https://github.com/apache/spark/pull/33810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903614004
**[Test build #142694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142694/testReport)** for PR 33810 at commit [`ff1353e`](https://github.com/apache/spark/commit/ff1353e4130ffaa83aee9c1134c0c1242abea2fc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904748345
thanks, merging to 3.1/3.0!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903652811
**[Test build #142695 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142695/testReport)** for PR 33810 at commit [`5b3fc53`](https://github.com/apache/spark/commit/5b3fc5395d1b75e14a9b23044c9e91ab6f4bd31a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903657461
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47197/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903614004
**[Test build #142694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142694/testReport)** for PR 33810 at commit [`ff1353e`](https://github.com/apache/spark/commit/ff1353e4130ffaa83aee9c1134c0c1242abea2fc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904668948
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142722/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903680240
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47197/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904539837
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47222/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903829523
ping @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903665833
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47196/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903644518
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47196/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903665773
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47196/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904667058
**[Test build #142722 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142722/testReport)** for PR 33810 at commit [`e633591`](https://github.com/apache/spark/commit/e633591c2b0d1a897c17206e7cc9138bf2cedb0b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903949248
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47206/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903871484
**[Test build #142695 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142695/testReport)** for PR 33810 at commit [`5b3fc53`](https://github.com/apache/spark/commit/5b3fc5395d1b75e14a9b23044c9e91ab6f4bd31a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904519872
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47222/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903943767
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142704/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903943717
**[Test build #142704 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142704/testReport)** for PR 33810 at commit [`1cbc87b`](https://github.com/apache/spark/commit/1cbc87b726a9df44c27b9b261639c5240e887753).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903889488
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142695/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904672994
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47228/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #33810:
URL: https://github.com/apache/spark/pull/33810#discussion_r694811604
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
##########
@@ -174,16 +175,36 @@ case class MapKeys(child: Expression)
""",
group = "array_funcs",
since = "2.4.0")
-case class ArraysZip(children: Seq[Expression]) extends Expression with ExpectsInputTypes {
+case class ArraysZip(children: Seq[Expression], names: Seq[Expression])
+ extends Expression with ExpectsInputTypes {
+
+ def this(children: Seq[Expression]) = {
+ this(
+ children,
+ children.zipWithIndex.map {
+ case (u: UnresolvedAttribute, _) => Literal(u.nameParts.last)
+ case (e: NamedExpression, _) if e.resolved => Literal(e.name)
+ case (e: NamedExpression, _) => NamePlaceholder
+ case (_, idx) => Literal(idx.toString)
+ })
+ }
+
+ if (children.size != names.size) {
+ throw new IllegalArgumentException(
+ "The numbers of zipped arrays and field names should be the same")
+ }
+
+ override lazy val resolved: Boolean =
+ childrenResolved && checkInputDataTypes().isSuccess && names.forall(_.resolved)
override def inputTypes: Seq[AbstractDataType] = Seq.fill(children.length)(ArrayType)
@transient override lazy val dataType: DataType = {
- val fields = children.zip(arrayElementTypes).zipWithIndex.map {
- case ((expr: NamedExpression, elementType), _) =>
- StructField(expr.name, elementType, nullable = true)
- case ((_, elementType), idx) =>
- StructField(idx.toString, elementType, nullable = true)
+ val fields = arrayElementTypes.zip(names).map {
+ case (elementType, Literal(name, StringType)) =>
+ StructField(name.toString, elementType, nullable = true)
+ case _ =>
+ throw new AnalysisException("Schema name of arrays_zip should be string literal.")
Review comment:
Right, done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903652811
**[Test build #142695 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142695/testReport)** for PR 33810 at commit [`5b3fc53`](https://github.com/apache/spark/commit/5b3fc5395d1b75e14a9b23044c9e91ab6f4bd31a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904673049
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47228/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903949227
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47206/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903949248
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47206/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904493370
**[Test build #142722 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142722/testReport)** for PR 33810 at commit [`e633591`](https://github.com/apache/spark/commit/e633591c2b0d1a897c17206e7cc9138bf2cedb0b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904673049
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47228/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-904539874
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47222/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33810:
URL: https://github.com/apache/spark/pull/33810#discussion_r694089641
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
##########
@@ -174,16 +174,35 @@ case class MapKeys(child: Expression)
""",
group = "array_funcs",
since = "2.4.0")
-case class ArraysZip(children: Seq[Expression]) extends Expression with ExpectsInputTypes {
+case class ArraysZip(children: Seq[Expression], names: Seq[Expression])
+ extends Expression with ExpectsInputTypes {
+
+ def this(children: Seq[Expression]) = {
+ this(
+ children,
+ children.zipWithIndex.map {
+ case (u: UnresolvedAttribute, _) => Literal(u.nameParts.last)
+ case (e: NamedExpression, _) if e.resolved => Literal(e.name)
+ case (e: NamedExpression, _) => NamePlaceholder
+ case (_, idx) => Literal(idx.toString)
+ })
+ }
+
+ if (children.size != names.size) {
+ throw new IllegalArgumentException(
+ "The numbers of zipped arrays and field names should be the same")
+ }
+
+ override lazy val resolved: Boolean =
+ childrenResolved && checkInputDataTypes().isSuccess && names.forall(_.resolved)
override def inputTypes: Seq[AbstractDataType] = Seq.fill(children.length)(ArrayType)
@transient override lazy val dataType: DataType = {
- val fields = children.zip(arrayElementTypes).zipWithIndex.map {
- case ((expr: NamedExpression, elementType), _) =>
- StructField(expr.name, elementType, nullable = true)
- case ((_, elementType), idx) =>
- StructField(idx.toString, elementType, nullable = true)
+ val fields = arrayElementTypes.zip(names).map {
+ case (elementType, Literal(name, StringType)) =>
+ StructField(name.toString, elementType, nullable = true)
+ case _ => StructField("", NullType)
Review comment:
why does 3.1 need this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33810: [SPARK-35876][SQL][3.1] ArraysZip should retain field names to avoid being re-written by analyzer/optimizer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33810:
URL: https://github.com/apache/spark/pull/33810#issuecomment-903939326
**[Test build #142704 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142704/testReport)** for PR 33810 at commit [`1cbc87b`](https://github.com/apache/spark/commit/1cbc87b726a9df44c27b9b261639c5240e887753).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org