You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2024/02/28 01:34:53 UTC
[PR] [SPARK-47199][PYTHON][TESTS] Add prefix into TemporaryDirectory to avoid flakiness [spark]
HyukjinKwon opened a new pull request, #45298:
URL: https://github.com/apache/spark/pull/45298
### What changes were proposed in this pull request?
This PR proposes to set `prefix` for `TemporaryDirectory` to deflake the tests. Sometimes the test fail because the temporary directory names are same (https://github.com/apache/spark/actions/runs/8066850485/job/22036007390).
```
File "/__w/spark/spark/python/pyspark/sql/dataframe.py", line ?, in pyspark.sql.dataframe.DataFrame.writeStream
Failed example:
with tempfile.TemporaryDirectory() as d:
# Create a table with Rate source.
df.writeStream.toTable(
"my_table", checkpointLocation=d)
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.11/doctest.py", line 1353, in __run
exec(compile(example.source, filename, "single",
File "<doctest pyspark.sql.dataframe.DataFrame.writeStream[3]>", line 1, in <module>
with tempfile.TemporaryDirectory() as d:
File "/usr/lib/python3.11/tempfile.py", line 1043, in __exit__
self.cleanup()
File "/usr/lib/python3.11/tempfile.py", line 1047, in cleanup
self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
File "/usr/lib/python3.11/tempfile.py", line 1029, in _rmtree
_rmtree(name, onerror=onerror)
File "/usr/lib/python3.11/shutil.py", line 738, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/usr/lib/python3.11/shutil.py", line 736, in rmtree
os.rmdir(path, dir_fd=dir_fd)
OSError: [Errno 39] Directory not empty: '/__w/spark/spark/python/target/4f062b09-213f-4ac2-a10a-2d704990141b/tmp29irqweq'
```
### Why are the changes needed?
To make the tests more robust.
### Does this PR introduce _any_ user-facing change?
No, test-only. There's a bit of user-facing documentation change but pretty trivial.
### How was this patch tested?
Manually tested. CI in this PR should test them out as well.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47199][PYTHON][TESTS] Add prefix into TemporaryDirectory to avoid flakiness [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #45298:
URL: https://github.com/apache/spark/pull/45298#discussion_r1505232292
##########
python/pyspark/sql/readwriter.py:
##########
@@ -400,7 +400,8 @@ def json(
Example 2: Read JSON from multiple files in a directory
>>> import tempfile
- >>> with tempfile.TemporaryDirectory() as d1, tempfile.TemporaryDirectory() as d2:
+ >>> with (tempfile.TemporaryDirectory(prefix="json2") as d1,
+ ... tempfile.TemporaryDirectory(prefix="json3") as d2):
Review Comment:
IIRC, the syntax:
```py
with (..., ...):
...
```
is not supported in Python 3.8.
It should be without `()`.
```py
with ..., ...:
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47199][PYTHON][TESTS] Add prefix into TemporaryDirectory to avoid flakiness [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #45298: [SPARK-47199][PYTHON][TESTS] Add prefix into TemporaryDirectory to avoid flakiness
URL: https://github.com/apache/spark/pull/45298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47199][PYTHON][TESTS] Add prefix into TemporaryDirectory to avoid flakiness [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45298:
URL: https://github.com/apache/spark/pull/45298#issuecomment-1968364450
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org