You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Mladjan Gadzic (Jira)" <ji...@apache.org> on 2023/05/16 07:51:00 UTC
[jira] [Comment Edited] (HDDS-7889) Intermittent failure in TestOzoneSnapshotRestore
[ https://issues.apache.org/jira/browse/HDDS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17723031#comment-17723031 ]
Mladjan Gadzic edited comment on HDDS-7889 at 5/16/23 7:50 AM:
---------------------------------------------------------------
Looking into this right now. Taking [https://github.com/mladjan-gadzic/ozone/actions/runs/4985395593] GitHub action as an example. From added logs [https://github.com/mladjan-gadzic/ozone/commits/HDDS-7889] it looks like *keyCopy* method either copies one more key, or copying of the key is not finished at the time when *keyCount* method is executed. Check logs:
{code:java}
2023-05-15 21:53:14,207 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:testRestoreSnapshot(227)) - ###Original key count###
2023-05-15 21:53:14,426 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=1, volumeName=vol-74060, bucketName=buc-54455, keyName=.snapshot/1173edc6-b0ae-47b6-8f7c-77e83e5d4821/key-0###
2023-05-15 21:53:14,426 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=2, volumeName=vol-74060, bucketName=buc-54455, keyName=.snapshot/1173edc6-b0ae-47b6-8f7c-77e83e5d4821/key-1###
2023-05-15 21:53:14,426 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=3, volumeName=vol-74060, bucketName=buc-54455, keyName=.snapshot/1173edc6-b0ae-47b6-8f7c-77e83e5d4821/key-2###
2023-05-15 21:53:14,427 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=4, volumeName=vol-74060, bucketName=buc-54455, keyName=.snapshot/1173edc6-b0ae-47b6-8f7c-77e83e5d4821/key-3###
2023-05-15 21:53:14,427 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=5, volumeName=vol-74060, bucketName=buc-54455, keyName=.snapshot/1173edc6-b0ae-47b6-8f7c-77e83e5d4821/key-4###
2023-05-15 21:53:14,526 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:testRestoreSnapshot(232)) - ###After deletion key count###
*all keys are deleted - no logs*
2023-05-15 21:53:15,896 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:testRestoreSnapshot(245)) - ###After snapshot key count###
2023-05-15 21:53:15,910 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=1, volumeName=vol-74060, bucketName=buc-54455, keyName=key-0###
2023-05-15 21:53:15,910 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=2, volumeName=vol-74060, bucketName=buc-54455, keyName=key-1###
2023-05-15 21:53:15,910 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=3, volumeName=vol-74060, bucketName=buc-54455, keyName=key-2###
2023-05-15 21:53:15,910 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=4, volumeName=vol-74060, bucketName=buc-54455, keyName=key-3###
2023-05-15 21:53:15,911 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=5, volumeName=vol-74060, bucketName=buc-54455, keyName=key-4###
2023-05-15 21:53:15,911 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=6, volumeName=vol-74060, bucketName=buc-54455, keyName=key-4._COPYING_###
{code}
was (Author: JIRAUSER298581):
Looking into this right now. Taking [https://github.com/mladjan-gadzic/ozone/actions/runs/4985395593] GitHub action as an example. From added logs [https://github.com/mladjan-gadzic/ozone/commits/HDDS-7889] it looks like *keyCopy* method either copies one more key, or copying of the key is not finished at the time when *keyCount* method is executed (only happens when FSO bucket layout is a source bucket). Check logs:
{code:java}
2023-05-15 21:53:14,207 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:testRestoreSnapshot(227)) - ###Original key count###
2023-05-15 21:53:14,426 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=1, volumeName=vol-74060, bucketName=buc-54455, keyName=.snapshot/1173edc6-b0ae-47b6-8f7c-77e83e5d4821/key-0###
2023-05-15 21:53:14,426 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=2, volumeName=vol-74060, bucketName=buc-54455, keyName=.snapshot/1173edc6-b0ae-47b6-8f7c-77e83e5d4821/key-1###
2023-05-15 21:53:14,426 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=3, volumeName=vol-74060, bucketName=buc-54455, keyName=.snapshot/1173edc6-b0ae-47b6-8f7c-77e83e5d4821/key-2###
2023-05-15 21:53:14,427 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=4, volumeName=vol-74060, bucketName=buc-54455, keyName=.snapshot/1173edc6-b0ae-47b6-8f7c-77e83e5d4821/key-3###
2023-05-15 21:53:14,427 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=5, volumeName=vol-74060, bucketName=buc-54455, keyName=.snapshot/1173edc6-b0ae-47b6-8f7c-77e83e5d4821/key-4###
2023-05-15 21:53:14,526 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:testRestoreSnapshot(232)) - ###After deletion key count###
*all keys are deleted - no logs*
2023-05-15 21:53:15,896 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:testRestoreSnapshot(245)) - ###After snapshot key count###
2023-05-15 21:53:15,910 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=1, volumeName=vol-74060, bucketName=buc-54455, keyName=key-0###
2023-05-15 21:53:15,910 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=2, volumeName=vol-74060, bucketName=buc-54455, keyName=key-1###
2023-05-15 21:53:15,910 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=3, volumeName=vol-74060, bucketName=buc-54455, keyName=key-2###
2023-05-15 21:53:15,910 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=4, volumeName=vol-74060, bucketName=buc-54455, keyName=key-3###
2023-05-15 21:53:15,911 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=5, volumeName=vol-74060, bucketName=buc-54455, keyName=key-4###
2023-05-15 21:53:15,911 [Listener at 127.0.0.1/35769] INFO snapshot.TestOzoneSnapshotRestore (TestOzoneSnapshotRestore.java:keyCount(186)) - ###keyNumber=6, volumeName=vol-74060, bucketName=buc-54455, keyName=key-4._COPYING_###
{code}
> Intermittent failure in TestOzoneSnapshotRestore
> ------------------------------------------------
>
> Key: HDDS-7889
> URL: https://issues.apache.org/jira/browse/HDDS-7889
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: test
> Affects Versions: 1.4.0
> Reporter: Attila Doroszlai
> Assignee: Mladjan Gadzic
> Priority: Major
>
> {code}
> org.apache.hadoop.ozone.om.snapshot.TestOzoneSnapshotRestore.testRestoreSnapshot(BucketLayout)[1] Time elapsed: 32.12 s <<< FAILURE!
> org.opentest4j.AssertionFailedError: expected: <5> but was: <6>
> ...
> at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:527)
> at org.apache.hadoop.ozone.om.snapshot.TestOzoneSnapshotRestore.testRestoreSnapshot(TestOzoneSnapshotRestore.java:232)
> {code}
> {code}
> org.apache.hadoop.ozone.om.snapshot.TestOzoneSnapshotRestore.testRestoreSnapshotDifferentBucket(BucketLayout)[2] Time elapsed: 26.708 s <<< FAILURE!
> org.opentest4j.AssertionFailedError: expected: <5> but was: <6>
> at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:527)
> at org.apache.hadoop.ozone.om.snapshot.TestOzoneSnapshotRestore.testRestoreSnapshotDifferentBucket(TestOzoneSnapshotRestore.java:290)
> {code}
> * https://github.com/adoroszlai/ozone-build-results/blob/master/2023/02/04/19941/it-om/hadoop-ozone/integration-test/org.apache.hadoop.ozone.om.snapshot.TestOzoneSnapshotRestore.txt
> * https://github.com/adoroszlai/ozone-build-results/blob/master/2023/05/07/22146/it-om/hadoop-ozone/integration-test/org.apache.hadoop.ozone.om.snapshot.TestOzoneSnapshotRestore.txt
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org