You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/09/28 07:50:20 UTC
[GitHub] [spark] LuciferYang opened a new pull request, #38028: [SPARK-40435][SQL][TESTS][FOLLOWUP] Correct test precondition of `PythonUDFSuite` and `ContinuousSuite`
LuciferYang opened a new pull request, #38028:
URL: https://github.com/apache/spark/pull/38028
### What changes were proposed in this pull request?
https://github.com/apache/spark/pull/37894 changed the preconditions for the following two tests from `assume(shouldTestGroupedAggPandasUDFs)` to `assume(shouldTestPythonUDFs)`:
- `SPARK-39962: Global aggregation of Pandas UDF should respect the column order` in `PythonUDFSuite`
- `continuous mode with various UDFs - Scalar Pandas UDF` in `ContinuousSuite`
but this change this change will cause test failure if `pandas` is not installed, so this pr restore the test preconditions from `assume(shouldTestPythonUDFs)` to `assume(shouldTestGroupedAggPandasUDFs)`.
### Why are the changes needed?
Fix test precondition of `PythonUDFSuite` and `ContinuousSuite`
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Actions
- Manual test, `pandas` is not installed:
```
build/sbt clean "sql/testOnly org.apache.spark.sql.execution.python.PythonUDFSuite"
build/sbt clean "sql/testOnly org.apache.spark.sql.streaming.continuous.ContinuousSuite"
```
Before
PythonUDFSuite
```
[info] - SPARK-39962: Global aggregation of Pandas UDF should respect the column order *** FAILED *** (799 milliseconds)
[info] java.lang.RuntimeException: Python executable [python3] and/or pyspark are unavailable.
[info] at org.apache.spark.sql.IntegratedUDFTestUtils$.pandasGroupedAggFunc$lzycompute(IntegratedUDFTestUtils.scala:236)
[info] at org.apache.spark.sql.IntegratedUDFTestUtils$.org$apache$spark$sql$IntegratedUDFTestUtils$$pandasGroupedAggFunc(IntegratedUDFTestUtils.scala:217)
[info] at org.apache.spark.sql.IntegratedUDFTestUtils$TestGroupedAggPandasUDF.udf$lzycompute(IntegratedUDFTestUtils.scala:433)
[info] at org.apache.spark.sql.IntegratedUDFTestUtils$TestGroupedAggPandasUDF.udf(IntegratedUDFTestUtils.scala:430)
[info] at org.apache.spark.sql.IntegratedUDFTestUtils$TestGroupedAggPandasUDF.apply(IntegratedUDFTestUtils.scala:444)
[info] at org.apache.spark.sql.execution.python.PythonUDFSuite.$anonfun$new$9(PythonUDFSuite.scala:82)
[info] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
```
and
ContinuousSuite
```
[info] - continuous mode with various UDFs - Scalar Pandas UDF *** FAILED *** (715 milliseconds)
[info] java.lang.RuntimeException: Python executable [python3] and/or pyspark are unavailable.
[info] at org.apache.spark.sql.IntegratedUDFTestUtils$.pandasFunc$lzycompute(IntegratedUDFTestUtils.scala:214)
[info] at org.apache.spark.sql.IntegratedUDFTestUtils$.org$apache$spark$sql$IntegratedUDFTestUtils$$pandasFunc(IntegratedUDFTestUtils.scala:194)
[info] at org.apache.spark.sql.IntegratedUDFTestUtils$TestScalarPandasUDF$$anon$2.<init>(IntegratedUDFTestUtils.scala:382)
[info] at org.apache.spark.sql.IntegratedUDFTestUtils$TestScalarPandasUDF.udf$lzycompute(IntegratedUDFTestUtils.scala:379)
[info] at org.apache.spark.sql.IntegratedUDFTestUtils$TestScalarPandasUDF.udf(IntegratedUDFTestUtils.scala:379)
[info] at org.apache.spark.sql.IntegratedUDFTestUtils$TestScalarPandasUDF.apply(IntegratedUDFTestUtils.scala:404)
[info] at org.apache.spark.sql.streaming.continuous.ContinuousSuite.$anonfun$new$24(ContinuousSuite.scala:289)
```
After
PythonUDFSuite
```
[info] Run completed in 11 seconds, 278 milliseconds.
[info] Total number of tests run: 4
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 4, failed 0, canceled 1, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 72 s (01:12), completed 2022-9-28 15:46:40
```
and
ContinuousSuite
```
[info] Run completed in 33 seconds, 197 milliseconds.
[info] Total number of tests run: 13
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 13, failed 0, canceled 1, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 64 s (01:04), completed 2022-9-28 15:49:45
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #38028: [SPARK-40435][SQL][SS][TESTS][FOLLOWUP] Correct test precondition of `PythonUDFSuite` and `ContinuousSuite`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #38028:
URL: https://github.com/apache/spark/pull/38028#issuecomment-1263047453
thanks @HyukjinKwon @HeartSaVioR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #38028: [SPARK-40435][SQL][SS][TESTS][FOLLOWUP] Correct test precondition of `PythonUDFSuite` and `ContinuousSuite`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38028:
URL: https://github.com/apache/spark/pull/38028#issuecomment-1261596254
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #38028: [SPARK-40435][SQL][TESTS][FOLLOWUP] Correct test precondition of `PythonUDFSuite` and `ContinuousSuite`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #38028:
URL: https://github.com/apache/spark/pull/38028#issuecomment-1260522666
cc @HeartSaVioR and @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #38028: [SPARK-40435][SQL][SS][TESTS][FOLLOWUP] Correct test precondition of `PythonUDFSuite` and `ContinuousSuite`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #38028: [SPARK-40435][SQL][SS][TESTS][FOLLOWUP] Correct test precondition of `PythonUDFSuite` and `ContinuousSuite`
URL: https://github.com/apache/spark/pull/38028
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org