You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/29 13:57:00 UTC
[GitHub] [spark] LuciferYang opened a new pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
LuciferYang opened a new pull request #29903:
URL: https://github.com/apache/spark/pull/29903
### What changes were proposed in this pull request?
After `SPARK-32851` set `CODEGEN_FACTORY_MODE` to `CODEGEN_ONLY` of `sparkConf` in `SharedSparkSessionBase` to construction `SparkSession` in test, the test suite `SPARK-32459: UDF should not fail on WrappedArray` in s.sql.UDFSuite exposed a codegen fallback issue in Scala 2.13 as follow:
```
- SPARK-32459: UDF should not fail on WrappedArray *** FAILED ***
Caused by: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 47, Column 99: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 47, Column 99: No applicable constructor/method found for zero actual parameters; candidates are: "public scala.collection.mutable.Builder scala.collection.mutable.ArraySeq$.newBuilder(java.lang.Object)", "public scala.collection.mutable.Builder scala.collection.mutable.ArraySeq$.newBuilder(scala.reflect.ClassTag)", "public abstract scala.collection.mutable.Builder scala.collection.EvidenceIterableFactory.newBuilder(java.lang.Object)"
```
The root cause is `WrappedArray` represent `mutable.ArraySeq` in Scala 2.13 and has a different constructor of `newBuilder` method.
The main change of is pr is add Scala 2.13 only code part to deal with `case match WrappedArray` in Scala 2.13.
### Why are the changes needed?
We need to support a Scala 2.13 build
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Scala 2.12: Pass the Jenkins or GitHub Action
- Scala 2.13: All tests passed.
Do the following:
```
dev/change-scala-version.sh 2.13
mvn clean install -DskipTests -pl sql/core -Pscala-2.13 -am
mvn test -pl sql/core -Pscala-2.13
```
**Before**
```
Tests: succeeded 8540, failed 1, canceled 1, ignored 52, pending 0
*** 1 TEST FAILED ***
```
**After**
```
Tests: succeeded 8541, failed 0, canceled 1, ignored 52, pending 0
All tests passed.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701596817
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701397102
**[Test build #129276 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129276/testReport)** for PR 29903 at commit [`9c36f6c`](https://github.com/apache/spark/commit/9c36f6cfc43eb2119bb302f7914d60e187c23986).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-700769967
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-702505111
thx ~ @srowen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701396396
This approach looks OK to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701149516
**[Test build #129261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129261/testReport)** for PR 29903 at commit [`9c36f6c`](https://github.com/apache/spark/commit/9c36f6cfc43eb2119bb302f7914d60e187c23986).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701202204
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129261/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701163671
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33878/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
srowen closed pull request #29903:
URL: https://github.com/apache/spark/pull/29903
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-700925303
**[Test build #129241 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129241/testReport)** for PR 29903 at commit [`a2e6d95`](https://github.com/apache/spark/commit/a2e6d95b76d20ad4c526504b54524e34b4ce8294).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701433573
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33893/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29903:
URL: https://github.com/apache/spark/pull/29903#discussion_r497238667
##########
File path: sql/catalyst/src/main/scala-2.12/org/apache/spark/sql/catalyst/expressions/objects/MapObjectsCodeGenHelper.scala
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.expressions.objects
+
+import scala.collection.mutable.{Builder, IndexedSeq, WrappedArray}
+
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext
+
+object MapObjectsCodeGenHelper {
Review comment:
Address 9c36f6c fix this comments, use `Properties.versionNumberString` to decide how to generate code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701163686
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701149516
**[Test build #129261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129261/testReport)** for PR 29903 at commit [`9c36f6c`](https://github.com/apache/spark/commit/9c36f6cfc43eb2119bb302f7914d60e187c23986).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-700926381
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701422878
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33893/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701885753
@srowen Jenkins test passed now :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-700769967
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701433590
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701596817
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701202198
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-700724916
**[Test build #129241 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129241/testReport)** for PR 29903 at commit [`a2e6d95`](https://github.com/apache/spark/commit/a2e6d95b76d20ad4c526504b54524e34b4ce8294).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701202198
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701397102
**[Test build #129276 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129276/testReport)** for PR 29903 at commit [`9c36f6c`](https://github.com/apache/spark/commit/9c36f6cfc43eb2119bb302f7914d60e187c23986).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-700769926
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33858/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #29903:
URL: https://github.com/apache/spark/pull/29903#discussion_r496745103
##########
File path: sql/catalyst/src/main/scala-2.12/org/apache/spark/sql/catalyst/expressions/objects/MapObjectsCodeGenHelper.scala
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.expressions.objects
+
+import scala.collection.mutable.{Builder, IndexedSeq, WrappedArray}
+
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext
+
+object MapObjectsCodeGenHelper {
Review comment:
Is it really necessary to have separate copies of this class? Perhaps just looking at the scala version or other properties of the byte code would let you decide on one or the other?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-700724916
**[Test build #129241 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129241/testReport)** for PR 29903 at commit [`a2e6d95`](https://github.com/apache/spark/commit/a2e6d95b76d20ad4c526504b54524e34b4ce8294).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-700723112
cc @srowen a new problem has been found of `sql/core` module in Scala 2.13
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701885753
@srowen now Jenkins test passed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701595818
**[Test build #129276 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129276/testReport)** for PR 29903 at commit [`9c36f6c`](https://github.com/apache/spark/commit/9c36f6cfc43eb2119bb302f7914d60e187c23986).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701433590
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29903:
URL: https://github.com/apache/spark/pull/29903#discussion_r497208048
##########
File path: sql/catalyst/src/main/scala-2.12/org/apache/spark/sql/catalyst/expressions/objects/MapObjectsCodeGenHelper.scala
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.expressions.objects
+
+import scala.collection.mutable.{Builder, IndexedSeq, WrappedArray}
+
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext
+
+object MapObjectsCodeGenHelper {
Review comment:
I think it is workable if we accept the use `Properties.versionNumberString.startsWith ("2.13")` condition to add `if else branch`branch, I will changed the code to this style first
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701157322
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33878/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-700757741
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33858/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-700926381
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701201839
**[Test build #129261 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129261/testReport)** for PR 29903 at commit [`9c36f6c`](https://github.com/apache/spark/commit/9c36f6cfc43eb2119bb302f7914d60e187c23986).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701396315
Jenkins retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-702140924
Merged to master
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29903:
URL: https://github.com/apache/spark/pull/29903#discussion_r497208048
##########
File path: sql/catalyst/src/main/scala-2.12/org/apache/spark/sql/catalyst/expressions/objects/MapObjectsCodeGenHelper.scala
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.expressions.objects
+
+import scala.collection.mutable.{Builder, IndexedSeq, WrappedArray}
+
+import org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext
+
+object MapObjectsCodeGenHelper {
Review comment:
I think it is workable if we accept the use `Properties.versionNumberString.startsWith ("2.13")` condition and add a `if else branch`, I will changed the code to this style first
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29903: [SPARK-33024][SQL] Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29903:
URL: https://github.com/apache/spark/pull/29903#issuecomment-701163686
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org