You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2020/07/25 10:08:43 UTC
[spark] branch master updated: [SPARK-32422][SQL][TESTS] Use
python3 executable instead of python3.6 in IntegratedUDFTestUtils
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 277a406 [SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils
277a406 is described below
commit 277a4063ef4ff6e91d999864b820607fa1a206aa
Author: HyukjinKwon <gu...@apache.org>
AuthorDate: Sat Jul 25 03:06:45 2020 -0700
[SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils
### What changes were proposed in this pull request?
This PR uses `python3` instead of `python3.6` executable as a fallback in `IntegratedUDFTestUtils`.
### Why are the changes needed?
Currently, GitHub Actions skips pandas UDFs. Python 3.8 is installed explicitly but somehow `python3.6` looks available in GitHub Actions build environment by default.
```
[info] - udf/postgreSQL/udf-case.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3.6]. !!! IGNORED !!!
...
[info] - udf/postgreSQL/udf-select_having.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3.6]. !!! IGNORED !!!
...
```
It was chosen as `python3.6` for Jenkins to pick one Python explicitly; however, looks we're already using `python3` here and there.
It will also reduce the overhead to fix when we deprecate or drop Python versions.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
It should be tested in Jenkins and GitHub Actions environments here.
Closes #29217 from HyukjinKwon/SPARK-32422.
Authored-by: HyukjinKwon <gu...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala b/sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala
index 6391d56..80346b3 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala
@@ -197,7 +197,7 @@ object IntegratedUDFTestUtils extends SQLHelper {
lazy val pythonExec: String = {
val pythonExec = sys.env.getOrElse(
- "PYSPARK_DRIVER_PYTHON", sys.env.getOrElse("PYSPARK_PYTHON", "python3.6"))
+ "PYSPARK_DRIVER_PYTHON", sys.env.getOrElse("PYSPARK_PYTHON", "python3"))
if (TestUtils.testCommandAvailable(pythonExec)) {
pythonExec
} else {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org