You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/09 09:08:28 UTC
[GitHub] [spark] attilapiros opened a new pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
attilapiros opened a new pull request #34233:
URL: https://github.com/apache/spark/pull/34233
### What changes were proposed in this pull request?
Extending the python test runner by logging out the temp output files.
### Why are the changes needed?
I was running a python test which was extremely slow and I was surprised the unit-tests.log has not been even created. Looked into the code and as I got the tests can be executed in parallel and each one has its own temporary output file which is only added to the unit-tests.log when a test is finished with a failure (after acquiring a lock to avoid parallel write on unit-tests.log).
To avoid such a confusion it would make sense to log out the path of those temporary output files this way when a test got stuck we can peek into its log file.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
I was running the python tests:
```
./python/run-tests
Running PySpark tests. Output is in /Users/attilazsoltpiros/git/attilapiros/spark/python/unit-tests.log
Will test against the following Python executables: ['/usr/local/bin/python3']
Will test the following Python modules: ['pyspark-core', 'pyspark-ml', 'pyspark-mllib', 'pyspark-pandas', 'pyspark-pandas-slow', 'pyspark-resource', 'pyspark-sql', 'pyspark-streaming']
/usr/local/bin/python3 python_implementation is CPython
/usr/local/bin/python3 version is: Python 3.9.7
Starting test(/usr/local/bin/python3): pyspark.ml.tests.test_feature (temp output: /var/folders/t_/fr_vqcyx23vftk81ftz1k5hw0000gn/T/usr_local_bin_python3__pyspark.ml.tests.test_feature__yc5_5mjk.log)
Starting test(/usr/local/bin/python3): pyspark.ml.tests.test_algorithms (temp output: /var/folders/t_/fr_vqcyx23vftk81ftz1k5hw0000gn/T/usr_local_bin_python3__pyspark.ml.tests.test_algorithms__icc6xxai.log)
Starting test(/usr/local/bin/python3): pyspark.ml.tests.test_base (temp output: /var/folders/t_/fr_vqcyx23vftk81ftz1k5hw0000gn/T/usr_local_bin_python3__pyspark.ml.tests.test_base__4m6xyiv5.log)
Starting test(/usr/local/bin/python3): pyspark.ml.tests.test_evaluation (temp output: /var/folders/t_/fr_vqcyx23vftk81ftz1k5hw0000gn/T/usr_local_bin_python3__pyspark.ml.tests.test_evaluation__fkzjlfmm.log)
Finished test(/usr/local/bin/python3): pyspark.ml.tests.test_base (16s)
Starting test(/usr/local/bin/python3): pyspark.ml.tests.test_image (temp output: /var/folders/t_/fr_vqcyx23vftk81ftz1k5hw0000gn/T/usr_local_bin_python3__pyspark.ml.tests.test_image__iuckk_c0.log)
Finished test(/usr/local/bin/python3): pyspark.ml.tests.test_evaluation (20s)
Starting test(/usr/local/bin/python3): pyspark.ml.tests.test_linalg (temp output: /var/folders/t_/fr_vqcyx23vftk81ftz1k5hw0000gn/T/usr_local_bin_python3__pyspark.ml.tests.test_linalg__3tncana4.log)
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-939269161
**[Test build #144053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144053/testReport)** for PR 34233 at commit [`b8bf885`](https://github.com/apache/spark/commit/b8bf885566c3d40990969074b755724df647662b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-939285020
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48530/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-940637934
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144117/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-939262478
**[Test build #144052 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144052/testReport)** for PR 34233 at commit [`017844f`](https://github.com/apache/spark/commit/017844fc6c1601e21b5d3d63004e3b13c531bb0a).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-939282823
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48530/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-939262366
**[Test build #144052 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144052/testReport)** for PR 34233 at commit [`017844f`](https://github.com/apache/spark/commit/017844fc6c1601e21b5d3d63004e3b13c531bb0a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-939269161
**[Test build #144053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144053/testReport)** for PR 34233 at commit [`b8bf885`](https://github.com/apache/spark/commit/b8bf885566c3d40990969074b755724df647662b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-939262487
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144052/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34233:
URL: https://github.com/apache/spark/pull/34233#discussion_r726674686
##########
File path: python/run-tests.py
##########
@@ -48,6 +48,11 @@ def print_red(text):
print('\033[31m' + text + '\033[0m')
+def get_valid_filename(s):
Review comment:
no biggy either but can we add a short comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-940666064
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48594/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-940637934
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144117/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-940615467
**[Test build #144117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144117/testReport)** for PR 34233 at commit [`0bf05ea`](https://github.com/apache/spark/commit/0bf05eabae1cf94169ec225c61b4d42bb40c7f39).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-939262487
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144052/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-939276934
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros closed pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
attilapiros closed pull request #34233:
URL: https://github.com/apache/spark/pull/34233
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-939262366
**[Test build #144052 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144052/testReport)** for PR 34233 at commit [`017844f`](https://github.com/apache/spark/commit/017844fc6c1601e21b5d3d63004e3b13c531bb0a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-939267994
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48529/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-940625992
**[Test build #144117 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144117/testReport)** for PR 34233 at commit [`0bf05ea`](https://github.com/apache/spark/commit/0bf05eabae1cf94169ec225c61b4d42bb40c7f39).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-940662436
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48594/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-939826180
cc @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-939273871
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48529/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-940615467
**[Test build #144117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144117/testReport)** for PR 34233 at commit [`0bf05ea`](https://github.com/apache/spark/commit/0bf05eabae1cf94169ec225c61b4d42bb40c7f39).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34233:
URL: https://github.com/apache/spark/pull/34233#discussion_r726674423
##########
File path: python/run-tests.py
##########
@@ -98,10 +103,12 @@ def run_individual_python_test(target_dir, test_name, pyspark_python):
]
env["PYSPARK_SUBMIT_ARGS"] = " ".join(spark_args)
- LOGGER.info("Starting test(%s): %s", pyspark_python, test_name)
+ output_prefix = get_valid_filename(pyspark_python + "__" + test_name + "__").lstrip("_")
+ per_test_output = tempfile.NamedTemporaryFile(prefix=output_prefix, suffix=".log")
+ LOGGER.info(
+ "Starting test(%s): %s (temp output: %s)", pyspark_python, test_name, per_test_output.name)
Review comment:
maybe just a matter of preference .. but maybe just printing out file name alone since it's pretty obvious .. ?
```suggestion
"Starting test(%s): %s (%s)", pyspark_python, test_name, per_test_output.name)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-940666064
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48594/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-939285020
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48530/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-939272968
**[Test build #144053 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144053/testReport)** for PR 34233 at commit [`b8bf885`](https://github.com/apache/spark/commit/b8bf885566c3d40990969074b755724df647662b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-939276934
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-939275054
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48530/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on a change in pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
attilapiros commented on a change in pull request #34233:
URL: https://github.com/apache/spark/pull/34233#discussion_r726717682
##########
File path: python/run-tests.py
##########
@@ -98,10 +103,12 @@ def run_individual_python_test(target_dir, test_name, pyspark_python):
]
env["PYSPARK_SUBMIT_ARGS"] = " ".join(spark_args)
- LOGGER.info("Starting test(%s): %s", pyspark_python, test_name)
+ output_prefix = get_valid_filename(pyspark_python + "__" + test_name + "__").lstrip("_")
+ per_test_output = tempfile.NamedTemporaryFile(prefix=output_prefix, suffix=".log")
+ LOGGER.info(
+ "Starting test(%s): %s (temp output: %s)", pyspark_python, test_name, per_test_output.name)
Review comment:
Well those files are really temporary files and when one test is finished it is deleted right away and I would like to avoid the same confusion I had with the `unit-tests.log`: to have the file name printed but the file does not exists (at least warn the reader about this).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-940557977
Looks fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34233:
URL: https://github.com/apache/spark/pull/34233#discussion_r726720048
##########
File path: python/run-tests.py
##########
@@ -98,10 +103,12 @@ def run_individual_python_test(target_dir, test_name, pyspark_python):
]
env["PYSPARK_SUBMIT_ARGS"] = " ".join(spark_args)
- LOGGER.info("Starting test(%s): %s", pyspark_python, test_name)
+ output_prefix = get_valid_filename(pyspark_python + "__" + test_name + "__").lstrip("_")
+ per_test_output = tempfile.NamedTemporaryFile(prefix=output_prefix, suffix=".log")
+ LOGGER.info(
+ "Starting test(%s): %s (temp output: %s)", pyspark_python, test_name, per_test_output.name)
Review comment:
👌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34233: [SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34233:
URL: https://github.com/apache/spark/pull/34233#issuecomment-940634316
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48594/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org