You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "manuzhang (via GitHub)" <gi...@apache.org> on 2024/07/29 16:07:50 UTC
[PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]
manuzhang opened a new pull request, #10811:
URL: https://github.com/apache/iceberg/pull/10811
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #10811:
URL: https://github.com/apache/iceberg/pull/10811#issuecomment-2343454242
cc @nastra help review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10811:
URL: https://github.com/apache/iceberg/pull/10811#discussion_r1695694630
##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWrites.java:
##########
@@ -419,5 +420,8 @@ public void testFaultToleranceOnWrite() throws IOException {
assertThat(snapshotBeforeFailingWrite).isEqualTo(snapshotAfterFailingWrite);
assertThat(resultBeforeFailingWrite).isEqualTo(resultAfterFailingWrite);
+
+ FileUtils.deleteDirectory(location);
Review Comment:
Shouldn't we just be using a temp directory fixture from Asserts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang closed pull request #10811: Spark 3.5: Fix flaky test due to deleting temp directory failure
URL: https://github.com/apache/iceberg/pull/10811
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10811:
URL: https://github.com/apache/iceberg/pull/10811#discussion_r1754328006
##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWrites.java:
##########
@@ -419,5 +421,13 @@ public void testFaultToleranceOnWrite() throws IOException {
assertThat(snapshotBeforeFailingWrite).isEqualTo(snapshotAfterFailingWrite);
assertThat(resultBeforeFailingWrite).isEqualTo(resultAfterFailingWrite);
+
+ while (location.exists()) {
Review Comment:
this seems like a workaround. https://stackoverflow.com/questions/56290320/junit-cannot-delete-tempdir-with-file-created-by-spark-structured-streaming indicates that the issue might be due to a resource not being closed, so maybe we should investigate what the actual root cause of this issue is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #10811:
URL: https://github.com/apache/iceberg/pull/10811#discussion_r1696191123
##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWrites.java:
##########
@@ -419,5 +420,8 @@ public void testFaultToleranceOnWrite() throws IOException {
assertThat(snapshotBeforeFailingWrite).isEqualTo(snapshotAfterFailingWrite);
assertThat(resultBeforeFailingWrite).isEqualTo(resultAfterFailingWrite);
+
+ FileUtils.deleteDirectory(location);
Review Comment:
The flakiness is because table data cannot be deleted sometimes. This PR ensures deleting table data before temp directory cleaning up itself.
> java.io.IOException: Failed to delete temp directory /tmp/junit8883793498240284997. The following paths could not be deleted (see suppressed exceptions for details): parquet/test/data/00001-127-9abadc06-9aa2-4b3b-8e77-66af8ddb7a3e-0-00001.parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on code in PR #10811:
URL: https://github.com/apache/iceberg/pull/10811#discussion_r1695524219
##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWrites.java:
##########
@@ -419,5 +420,8 @@ public void testFaultToleranceOnWrite() throws IOException {
assertThat(snapshotBeforeFailingWrite).isEqualTo(snapshotAfterFailingWrite);
assertThat(resultBeforeFailingWrite).isEqualTo(resultAfterFailingWrite);
+
+ FileUtils.deleteDirectory(location);
Review Comment:
thank you @manuzhang , can you expand a bit on how this fix the flaky tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #10811:
URL: https://github.com/apache/iceberg/pull/10811#issuecomment-2316566447
@Fokko @dramaticlly PTAL. This PR now proactively deletes temporary directory catching exceptions until it's cleaned up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #10811:
URL: https://github.com/apache/iceberg/pull/10811
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #10811:
URL: https://github.com/apache/iceberg/pull/10811#discussion_r1696191123
##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWrites.java:
##########
@@ -419,5 +420,8 @@ public void testFaultToleranceOnWrite() throws IOException {
assertThat(snapshotBeforeFailingWrite).isEqualTo(snapshotAfterFailingWrite);
assertThat(resultBeforeFailingWrite).isEqualTo(resultAfterFailingWrite);
+
+ FileUtils.deleteDirectory(location);
Review Comment:
> java.io.IOException: Failed to delete temp directory /tmp/junit8883793498240284997. The following paths could not be deleted (see suppressed exceptions for details): parquet/test/data/00001-127-9abadc06-9aa2-4b3b-8e77-66af8ddb7a3e-0-00001.parquet
The flakiness is because table data cannot be deleted sometimes. This PR ensures deleting table data before temp directory cleaning up itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang closed pull request #10811: Spark 3.5: Fix flaky test due to deleting temp directory failure
URL: https://github.com/apache/iceberg/pull/10811
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #10811:
URL: https://github.com/apache/iceberg/pull/10811#discussion_r1696191123
##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWrites.java:
##########
@@ -419,5 +420,8 @@ public void testFaultToleranceOnWrite() throws IOException {
assertThat(snapshotBeforeFailingWrite).isEqualTo(snapshotAfterFailingWrite);
assertThat(resultBeforeFailingWrite).isEqualTo(resultAfterFailingWrite);
+
+ FileUtils.deleteDirectory(location);
Review Comment:
The flakiness is because table data cannot be deleted sometimes. This PR ensures deleting table data before temp directory cleaning up itself.
```
java.io.IOException: Failed to delete temp directory /tmp/junit8883793498240284997. The following paths could not be deleted (see suppressed exceptions for details): parquet/test/data/00001-127-9abadc06-9aa2-4b3b-8e77-66af8ddb7a3e-0-00001.parquet
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang closed pull request #10811: Spark 3.5: Fix flaky test due to deleting temp directory failure
URL: https://github.com/apache/iceberg/pull/10811
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang closed pull request #10811: Spark 3.5: Fix flaky test due to deleting temp directory failure
URL: https://github.com/apache/iceberg/pull/10811
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark 3.5: Fix flaky test due to deleting temp directory failure [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #10811:
URL: https://github.com/apache/iceberg/pull/10811#issuecomment-2257733586
This PR didn't fix but exposed more underlying errors from https://github.com/apache/iceberg/actions/runs/10156307633/job/28084490263?pr=10811
```
TestDataFrameWrites > testFaultToleranceOnWrite() > format = avro FAILED
org.apache.commons.io.IOExceptionList: 1 exception(s): [org.apache.commons.io.IOIndexedException: IOException #1: Cannot delete file: /tmp/junit60135975027943685/parquet/test/data]
at org.apache.commons.io.IOExceptionList.checkEmpty(IOExceptionList.java:49)
at org.apache.commons.io.function.IOStream.forAll(IOStream.java:352)
at org.apache.commons.io.function.IOStreams.forAll(IOStreams.java:42)
at org.apache.commons.io.function.IOStreams.forAll(IOStreams.java:36)
at org.apache.commons.io.function.IOConsumer.forAll(IOConsumer.java:80)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:333)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1192)
at org.apache.iceberg.spark.source.TestDataFrameWrites.testFaultToleranceOnWrite(TestDataFrameWrites.java:424)
Caused by:
org.apache.commons.io.IOIndexedException: IOException #1: Cannot delete file: /tmp/junit60135975027943685/parquet/test/data
at org.apache.commons.io.function.IOStream.lambda$forAll$11(IOStream.java:347)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
at org.apache.commons.io.function.IOStream.forAll(IOStream.java:338)
... 6 more
Caused by:
java.io.IOException: Cannot delete file: /tmp/junit60135975027943685/parquet/test/data
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1340)
at org.apache.commons.io.function.IOStream.lambda$forAll$11(IOStream.java:340)
... 9 more
Caused by:
java.nio.file.NoSuchFileException: /tmp/junit60135975027943685/parquet/test/data/.00008-134-fed69c74-d8da-41c3-937f-ea0ab632d7d0-0-00001.parquet.crc
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.nio.file.Files.readAttributes(Files.java:1737)
at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)
at java.nio.file.Files.walkFileTree(Files.java:2706)
at java.nio.file.Files.walkFileTree(Files.java:2742)
at org.apache.commons.io.file.PathUtils.visitFileTree(PathUtils.java:1654)
at org.apache.commons.io.file.PathUtils.deleteDirectory(PathUtils.java:517)
at org.apache.commons.io.file.PathUtils.delete(PathUtils.java:476)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1337)
... 10 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org