You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/07/26 05:37:20 UTC
[GitHub] [spark] dongjoon-hyun opened a new pull request #29241: [SPARK-32443][CORE] Use POSIX compatible `command -v` in testCommandAvailable
dongjoon-hyun opened a new pull request #29241:
URL: https://github.com/apache/spark/pull/29241
### What changes were proposed in this pull request?
Currently, Apache Spark literally executes a given command to check the existence. This is dangerous and doesn't work sometimes. This PR aims to use `command -v`. `command` is POSIX-compatible and a light-weight and safe way to check the availability.
- **POSIX.1-2017**: `command` https://pubs.opengroup.org/onlinepubs/9699919799/
### Why are the changes needed?
```scala
scala> sys.process.Process("cat").run().exitValue()
res0: Int = 0
scala> sys.process.Process("ls").run().exitValue()
LICENSE
NOTICE
bin
doc
lib
man
res1: Int = 0
scala> sys.process.Process("rm").run().exitValue()
usage: rm [-f | -i] [-dPRrvW] file ...
unlink file
res4: Int = 64
```
```
scala> sys.process.Process("command -v rm").run().exitValue()
/bin/rm
res5: Int = 0
```
### Does this PR introduce _any_ user-facing change?
No. Although this is inside `main` source directory, this is used for testing purpose.
### How was this patch tested?
Pass the Jenkins with the existing tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663939283
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29241:
URL: https://github.com/apache/spark/pull/29241#discussion_r460570073
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
##########
@@ -28,6 +28,8 @@ class PythonUDFSuite extends QueryTest with SharedSparkSession {
val scalaTestUDF = TestScalaUDF(name = "scalaUDF")
val pythonTestUDF = TestPythonUDF(name = "pyUDF")
+ assume(pythonExec.equals("python3"))
Review comment:
For now, let me override this simply.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664019466
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664050723
**[Test build #126587 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126587/testReport)** for PR 29241 at commit [`c31277a`](https://github.com/apache/spark/commit/c31277a125b9d7539ee68b89dd5e08c79f4d4ab2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664064944
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664021199
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664038431
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664161752
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126613/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664049771
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664037656
**[Test build #126580 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126580/testReport)** for PR 29241 at commit [`b768a21`](https://github.com/apache/spark/commit/b768a218bb6a6927b43bbfca06ea894861aa82f6).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664162865
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664019309
**[Test build #126577 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126577/testReport)** for PR 29241 at commit [`fad9662`](https://github.com/apache/spark/commit/fad9662bb76580c9f97bfba5e9ecc4d53795bd19).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664166354
**[Test build #126623 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126623/testReport)** for PR 29241 at commit [`d239ede`](https://github.com/apache/spark/commit/d239eded8f854a1d0fc30bac12f1f5c080af02eb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664162865
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29241:
URL: https://github.com/apache/spark/pull/29241#discussion_r460682607
##########
File path: core/src/main/scala/org/apache/spark/TestUtils.scala
##########
@@ -236,7 +236,11 @@ private[spark] object TestUtils {
* Test if a command is available.
*/
def testCommandAvailable(command: String): Boolean = {
- val attempt = Try(Process(command).run(ProcessLogger(_ => ())).exitValue())
+ val attempt = if (Utils.isWindows) {
+ Try(Process(command).run(ProcessLogger(_ => ())).exitValue())
Review comment:
```suggestion
Try(Process(s"WHERE $command").run(ProcessLogger(_ => ())).exitValue())
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664031752
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664039221
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29241:
URL: https://github.com/apache/spark/pull/29241#discussion_r460569961
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
##########
@@ -28,6 +28,8 @@ class PythonUDFSuite extends QueryTest with SharedSparkSession {
val scalaTestUDF = TestScalaUDF(name = "scalaUDF")
val pythonTestUDF = TestPythonUDF(name = "pyUDF")
+ assume(pythonExec.equals("python3"))
Review comment:
Technically, we hit the similar issue before at SPARK-32422 two days ago. cc @HyukjinKwon .
```
[SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils
```
```scala
lazy val pythonExec: String = {
val pythonExec = sys.env.getOrElse(
"PYSPARK_DRIVER_PYTHON", sys.env.getOrElse("PYSPARK_PYTHON", "python3"))
if (TestUtils.testCommandAvailable(pythonExec)) {
pythonExec
} else {
"python"
}
}
```
At SPARK-32422, `python3.6` doesn't work correctly and @HyukjinKwon replaced it from `python3.6` to `python3`. The root cause is that we have multiple Python installation environment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664064944
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664125742
**[Test build #126613 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126613/testReport)** for PR 29241 at commit [`acf87a2`](https://github.com/apache/spark/commit/acf87a2289133a6292835eb9cd8700ccb7460da9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664039088
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663947676
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664165280
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664162872
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126608/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664145206
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29241:
URL: https://github.com/apache/spark/pull/29241#discussion_r460619887
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala
##########
@@ -200,8 +205,11 @@ object IntegratedUDFTestUtils extends SQLHelper {
"PYSPARK_DRIVER_PYTHON", sys.env.getOrElse("PYSPARK_PYTHON", "python3"))
if (TestUtils.testCommandAvailable(pythonExec)) {
pythonExec
+ } else if (pythonExec.equalsIgnoreCase("python") &&
Review comment:
@dongjoon-hyun, so just to clarify, can the changes in this file be removed because Jenkins uses `python` and GitHun Actions uses `python3` for now, and the tests pass correctly with the use of `command -v`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664037661
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664049667
**[Test build #126581 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126581/testReport)** for PR 29241 at commit [`d1f7145`](https://github.com/apache/spark/commit/d1f714566d37fbe91545f210ea3e0cb0d0e42d42).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663938680
**[Test build #126562 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126562/testReport)** for PR 29241 at commit [`7d561d7`](https://github.com/apache/spark/commit/7d561d73504fbf804a41f6fc390b037a5833fc14).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664019466
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664049771
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664039099
**[Test build #126581 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126581/testReport)** for PR 29241 at commit [`d1f7145`](https://github.com/apache/spark/commit/d1f714566d37fbe91545f210ea3e0cb0d0e42d42).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664147495
**[Test build #126619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126619/testReport)** for PR 29241 at commit [`0bf0773`](https://github.com/apache/spark/commit/0bf07732ee5a0ef278f5823fe46873c1c0249f94).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664166831
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29241:
URL: https://github.com/apache/spark/pull/29241#discussion_r460689868
##########
File path: core/src/main/scala/org/apache/spark/TestUtils.scala
##########
@@ -236,7 +236,11 @@ private[spark] object TestUtils {
* Test if a command is available.
*/
def testCommandAvailable(command: String): Boolean = {
- val attempt = Try(Process(command).run(ProcessLogger(_ => ())).exitValue())
+ val attempt = if (Utils.isWindows) {
+ Try(Process(s"WHERE $command").run(ProcessLogger(_ => ())).exitValue())
Review comment:
I run some tests via AppVeyor manually to show it works in Windows:
Build started: [CORE] `org.apache.spark.rdd.PipedRDDSuite` [](https://ci.appveyor.com/project/HyukjinKwon/spark/branch/5D8964EE-6046-4DBA-91AD-53B2E5EE4500)
Build started: [SQL] `org.apache.spark.sql.hive.execution.SQLQuerySuite` [](https://ci.appveyor.com/project/HyukjinKwon/spark/branch/C22F6129-6D78-40AF-BE21-1A625BC656A1)
Some of tests might already fail. We can just see if `testCommandAvailable` works or not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664037558
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664705415
Thank you @dongjoon-hyun and @srowen.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664066688
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664091874
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664039099
**[Test build #126581 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126581/testReport)** for PR 29241 at commit [`d1f7145`](https://github.com/apache/spark/commit/d1f714566d37fbe91545f210ea3e0cb0d0e42d42).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663964209
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664146047
I'll continue tomorrow because this should pass in both Jenkins/GitHub Action environment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663939206
**[Test build #126563 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126563/testReport)** for PR 29241 at commit [`dc74150`](https://github.com/apache/spark/commit/dc7415031fde4d1b423d6ac4b2e5f7c75227aa78).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29241:
URL: https://github.com/apache/spark/pull/29241#discussion_r460682607
##########
File path: core/src/main/scala/org/apache/spark/TestUtils.scala
##########
@@ -236,7 +236,11 @@ private[spark] object TestUtils {
* Test if a command is available.
*/
def testCommandAvailable(command: String): Boolean = {
- val attempt = Try(Process(command).run(ProcessLogger(_ => ())).exitValue())
+ val attempt = if (Utils.isWindows) {
+ Try(Process(command).run(ProcessLogger(_ => ())).exitValue())
Review comment:
```suggestion
Try(Process($"WHERE $command").run(ProcessLogger(_ => ())).exitValue())
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664110196
Thank you, @HyukjinKwon !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664037664
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126580/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664037412
**[Test build #126580 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126580/testReport)** for PR 29241 at commit [`b768a21`](https://github.com/apache/spark/commit/b768a218bb6a6927b43bbfca06ea894861aa82f6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664064949
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126587/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664064689
**[Test build #126587 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126587/testReport)** for PR 29241 at commit [`c31277a`](https://github.com/apache/spark/commit/c31277a125b9d7539ee68b89dd5e08c79f4d4ab2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664031991
It turns out that `python` is used in Jenkins. Let me take a look at that.
```
org.scalatest.exceptions.TestCanceledException: org.apache.spark.sql.IntegratedUDFTestUtils.pythonExec.equals("python3") was false
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29241:
URL: https://github.com/apache/spark/pull/29241#discussion_r460689868
##########
File path: core/src/main/scala/org/apache/spark/TestUtils.scala
##########
@@ -236,7 +236,11 @@ private[spark] object TestUtils {
* Test if a command is available.
*/
def testCommandAvailable(command: String): Boolean = {
- val attempt = Try(Process(command).run(ProcessLogger(_ => ())).exitValue())
+ val attempt = if (Utils.isWindows) {
+ Try(Process(s"WHERE $command").run(ProcessLogger(_ => ())).exitValue())
Review comment:
I run some tests via AppVeyor manually to show it works in Windows:
Build started: [CORE] `org.apache.spark.rdd.PipedRDDSuite` [](https://ci.appveyor.com/project/HyukjinKwon/spark/branch/5D8964EE-6046-4DBA-91AD-53B2E5EE4500)
Build started: [SQL] `org.apache.spark.sql.hive.execution.SQLQuerySuite` [](https://ci.appveyor.com/project/HyukjinKwon/spark/branch/C22F6129-6D78-40AF-BE21-1A625BC656A1)
Some of tests might already fail. We can just see if `testCommandAvailable` works or not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663951448
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664091877
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126597/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664031622
**[Test build #126577 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126577/testReport)** for PR 29241 at commit [`fad9662`](https://github.com/apache/spark/commit/fad9662bb76580c9f97bfba5e9ecc4d53795bd19).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663964209
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664039221
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663947676
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664021047
**[Test build #126575 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126575/testReport)** for PR 29241 at commit [`7907695`](https://github.com/apache/spark/commit/7907695acc03bc08a7e35a724582f2f368a97158).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664161093
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664002468
**[Test build #126575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126575/testReport)** for PR 29241 at commit [`7907695`](https://github.com/apache/spark/commit/7907695acc03bc08a7e35a724582f2f368a97158).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663951448
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664037661
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663938749
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29241:
URL: https://github.com/apache/spark/pull/29241#discussion_r460637686
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala
##########
@@ -200,8 +205,11 @@ object IntegratedUDFTestUtils extends SQLHelper {
"PYSPARK_DRIVER_PYTHON", sys.env.getOrElse("PYSPARK_PYTHON", "python3"))
if (TestUtils.testCommandAvailable(pythonExec)) {
pythonExec
+ } else if (pythonExec.equalsIgnoreCase("python") &&
Review comment:
Sure!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663938749
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663939283
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664126178
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664002639
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664238519
**[Test build #126623 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126623/testReport)** for PR 29241 at commit [`d239ede`](https://github.com/apache/spark/commit/d239eded8f854a1d0fc30bac12f1f5c080af02eb).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29241:
URL: https://github.com/apache/spark/pull/29241#discussion_r460682607
##########
File path: core/src/main/scala/org/apache/spark/TestUtils.scala
##########
@@ -236,7 +236,11 @@ private[spark] object TestUtils {
* Test if a command is available.
*/
def testCommandAvailable(command: String): Boolean = {
- val attempt = Try(Process(command).run(ProcessLogger(_ => ())).exitValue())
+ val attempt = if (Utils.isWindows) {
+ Try(Process(command).run(ProcessLogger(_ => ())).exitValue())
Review comment:
```suggestion
Try(Process(s"WHERE $command").run(ProcessLogger(_ => ())).exitValue())
```
##########
File path: core/src/main/scala/org/apache/spark/TestUtils.scala
##########
@@ -236,7 +236,11 @@ private[spark] object TestUtils {
* Test if a command is available.
*/
def testCommandAvailable(command: String): Boolean = {
- val attempt = Try(Process(command).run(ProcessLogger(_ => ())).exitValue())
+ val attempt = if (Utils.isWindows) {
+ Try(Process(command).run(ProcessLogger(_ => ())).exitValue())
Review comment:
@dongjoon-hyun, I manually tested. `command` -> `$"WHERE $command"` seems working fine in Windows with Scala 2.13.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29241:
URL: https://github.com/apache/spark/pull/29241#discussion_r460681180
##########
File path: core/src/main/scala/org/apache/spark/TestUtils.scala
##########
@@ -236,7 +236,11 @@ private[spark] object TestUtils {
* Test if a command is available.
*/
def testCommandAvailable(command: String): Boolean = {
- val attempt = Try(Process(command).run(ProcessLogger(_ => ())).exitValue())
+ val attempt = if (Utils.isWindows) {
+ Try(Process(command).run(ProcessLogger(_ => ())).exitValue())
Review comment:
@dongjoon-hyun, I manually tested. `command` -> `$"WHERE $command"` seems working fine in Windows with Scala 2.13.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664161740
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664049774
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126581/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664037558
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664029969
For now, `IntegratedUDFTestUtils.isPythonAvailable` function seems to work differently.
- In GitHub Action environment, it returns `true`.
- In Jenkins environment, it returns `false`.
```
IntegratedUDFTestUtils.isPythonAvailable was false
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663964073
**[Test build #126565 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126565/testReport)** for PR 29241 at commit [`dc74150`](https://github.com/apache/spark/commit/dc7415031fde4d1b423d6ac4b2e5f7c75227aa78).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664050723
**[Test build #126587 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126587/testReport)** for PR 29241 at commit [`c31277a`](https://github.com/apache/spark/commit/c31277a125b9d7539ee68b89dd5e08c79f4d4ab2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663998407
Yes. Everything is `test`.
```
$ git grep testCommandAvailable | grep -v 'def testCommandAvailable'
core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala: assume(TestUtils.testCommandAvailable("cat"))
core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala: assume(TestUtils.testCommandAvailable("wc"))
core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala: assume(TestUtils.testCommandAvailable("cat"))
core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala: assume(TestUtils.testCommandAvailable("cat"))
core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala: assume(TestUtils.testCommandAvailable("cat"))
core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala: assume(TestUtils.testCommandAvailable(envCommand))
core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala: assume(!TestUtils.testCommandAvailable("some_nonexistent_command"))
core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala: assume(TestUtils.testCommandAvailable("cat"))
core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala: assume(TestUtils.testCommandAvailable("cat"))
core/src/test/scala/org/apache/spark/rdd/PipedRDDSuite.scala: assume(TestUtils.testCommandAvailable(envCommand))
sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala: private lazy val isPythonAvailable: Boolean = TestUtils.testCommandAvailable(pythonExec)
sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala: if (TestUtils.testCommandAvailable(pythonExec)) {
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala: skip = !TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala: skip = !TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala: skip = !TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala: skip = !TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala: skip = !TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala: skip = !TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala: assume(TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala: skip = !TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala: skip = !TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala: assume(TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala: assume(TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala: assume(TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala: assume(TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala: assume(TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala: assume(TestUtils.testCommandAvailable("python"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala: assume(TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala: assume(TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala: assume(TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala: assume(TestUtils.testCommandAvailable("echo | sed"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala: assume(TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala: assume(TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala: assume(TestUtils.testCommandAvailable("/bin/bash"))
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala: assume(TestUtils.testCommandAvailable("/bin/bash"))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663947703
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126562/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664066524
**[Test build #126597 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126597/testReport)** for PR 29241 at commit [`6f5e9da`](https://github.com/apache/spark/commit/6f5e9da06db644866f071ead74bae00445190092).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664038431
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664091874
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664002639
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663947548
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664161361
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29241:
URL: https://github.com/apache/spark/pull/29241#discussion_r460569961
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
##########
@@ -28,6 +28,8 @@ class PythonUDFSuite extends QueryTest with SharedSparkSession {
val scalaTestUDF = TestScalaUDF(name = "scalaUDF")
val pythonTestUDF = TestPythonUDF(name = "pyUDF")
+ assume(pythonExec.equals("python3"))
Review comment:
Technically, we hit the similar issue before at SPARK-32422 two days ago. cc @HyukjinKwon .
```
[SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils
```
```
lazy val pythonExec: String = {
val pythonExec = sys.env.getOrElse(
"PYSPARK_DRIVER_PYTHON", sys.env.getOrElse("PYSPARK_PYTHON", "python3"))
if (TestUtils.testCommandAvailable(pythonExec)) {
pythonExec
} else {
"python"
}
}
```
At SPARK-32422, `python3.6` doesn't work correctly and @HyukjinKwon replaced it from `python3.6` to `python3`. The root cause is that we have multiple Python installation environment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664111261
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664039088
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #29241:
URL: https://github.com/apache/spark/pull/29241#discussion_r460568545
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
##########
@@ -28,6 +28,8 @@ class PythonUDFSuite extends QueryTest with SharedSparkSession {
val scalaTestUDF = TestScalaUDF(name = "scalaUDF")
val pythonTestUDF = TestPythonUDF(name = "pyUDF")
+ assume(pythonExec.equals("python3"))
Review comment:
If this is the issue, assert that it's either python or python3? the python 3 binary could be just python.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664109865
The PR is rebased to the master and reverted to the first commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29241:
URL: https://github.com/apache/spark/pull/29241#discussion_r460689868
##########
File path: core/src/main/scala/org/apache/spark/TestUtils.scala
##########
@@ -236,7 +236,11 @@ private[spark] object TestUtils {
* Test if a command is available.
*/
def testCommandAvailable(command: String): Boolean = {
- val attempt = Try(Process(command).run(ProcessLogger(_ => ())).exitValue())
+ val attempt = if (Utils.isWindows) {
+ Try(Process(s"WHERE $command").run(ProcessLogger(_ => ())).exitValue())
Review comment:
I run some tests via AppVeyor manually to show it works in Windows:
Build started: [CORE] `org.apache.spark.rdd.PipedRDDSuite` [](https://ci.appveyor.com/project/HyukjinKwon/spark/branch/26712025-86F9-40C2-BF08-1E4BD683B021)
Build started: [SQL] `org.apache.spark.sql.hive.execution.SQLQuerySuite` [](https://ci.appveyor.com/project/HyukjinKwon/spark/branch/07527AA5-8FF9-4E05-BA66-788B54BD8A26)
Some of tests might already fail. We can just see if `testCommandAvailable` works or not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664161356
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663964217
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126565/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663951367
**[Test build #126565 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126565/testReport)** for PR 29241 at commit [`dc74150`](https://github.com/apache/spark/commit/dc7415031fde4d1b423d6ac4b2e5f7c75227aa78).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664126718
Sorry guys. I updated this PR once more to add a fallback for `Windows`
- `cmd` doesn't support `command`.
- `PowerShell` has `command`, but `-v` is not supported.
For Windows, let's revisit after we get a complete run on Linux first.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664021199
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664166354
**[Test build #126623 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126623/testReport)** for PR 29241 at commit [`d239ede`](https://github.com/apache/spark/commit/d239eded8f854a1d0fc30bac12f1f5c080af02eb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664031752
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664126718
Sorry guys. I updated this PR once more to add a fallback for `Windows`
- `cmd` doesn't support `command`.
- `PowerShell` has `command`, but `-v` is not supported.
For Windows, let's revisit after we get a complete run on Linux first. PR description is also updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663939972
Hi, @srowen .
This PR will fix `PipedRDDSuite` hang situation at Scala 2.13, but I'd like to have this fix in older branches, too.
```
$ git grep testCommandAvailable | grep -v 'def testCommandAvailable' | wc -l
36
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664126178
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664091573
**[Test build #126597 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126597/testReport)** for PR 29241 at commit [`6f5e9da`](https://github.com/apache/spark/commit/6f5e9da06db644866f071ead74bae00445190092).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664019309
**[Test build #126577 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126577/testReport)** for PR 29241 at commit [`fad9662`](https://github.com/apache/spark/commit/fad9662bb76580c9f97bfba5e9ecc4d53795bd19).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663947698
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663938680
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664066688
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664239439
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664239439
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664066524
**[Test build #126597 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126597/testReport)** for PR 29241 at commit [`6f5e9da`](https://github.com/apache/spark/commit/6f5e9da06db644866f071ead74bae00445190092).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663951281
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664239449
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126623/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664166831
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664031753
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126577/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664002468
**[Test build #126575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126575/testReport)** for PR 29241 at commit [`7907695`](https://github.com/apache/spark/commit/7907695acc03bc08a7e35a724582f2f368a97158).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-663951367
**[Test build #126565 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126565/testReport)** for PR 29241 at commit [`dc74150`](https://github.com/apache/spark/commit/dc7415031fde4d1b423d6ac4b2e5f7c75227aa78).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664037412
**[Test build #126580 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126580/testReport)** for PR 29241 at commit [`b768a21`](https://github.com/apache/spark/commit/b768a218bb6a6927b43bbfca06ea894861aa82f6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664111012
**[Test build #126608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126608/testReport)** for PR 29241 at commit [`46abc40`](https://github.com/apache/spark/commit/46abc40c45e70dd495d7b9aa883a1f05b8566c7c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664111261
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29241:
URL: https://github.com/apache/spark/pull/29241#discussion_r460570073
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
##########
@@ -28,6 +28,8 @@ class PythonUDFSuite extends QueryTest with SharedSparkSession {
val scalaTestUDF = TestScalaUDF(name = "scalaUDF")
val pythonTestUDF = TestPythonUDF(name = "pyUDF")
+ assume(pythonExec.equals("python3"))
Review comment:
For now, let me override this simply for testing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664021200
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126575/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664145206
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664031991
It turns out that
- `python` is used in Jenkins.
- `python3` is used in GitHub Action.
```
org.scalatest.exceptions.TestCanceledException: org.apache.spark.sql.IntegratedUDFTestUtils.pythonExec.equals("python3") was false
```
Let me take a look at that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664050856
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664161356
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664125742
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29241:
URL: https://github.com/apache/spark/pull/29241#discussion_r460569303
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
##########
@@ -28,6 +28,8 @@ class PythonUDFSuite extends QueryTest with SharedSparkSession {
val scalaTestUDF = TestScalaUDF(name = "scalaUDF")
val pythonTestUDF = TestPythonUDF(name = "pyUDF")
+ assume(pythonExec.equals("python3"))
Review comment:
Since we drop Python2 officially, I'm looking this problem in more broader perspective.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29241: [SPARK-32443][CORE] Use POSIX-compatible `command -v` in testCommandAvailable
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29241:
URL: https://github.com/apache/spark/pull/29241#issuecomment-664050856
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org