You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Christos Bisias (Jira)" <ji...@apache.org> on 2024/01/15 16:55:00 UTC
[jira] [Comment Edited] (HDDS-10059) [disabled] Intermittent failure in TestOMRatisSnapshots.testInstallSnapshot
[ https://issues.apache.org/jira/browse/HDDS-10059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17806913#comment-17806913 ]
Christos Bisias edited comment on HDDS-10059 at 1/15/24 4:54 PM:
-----------------------------------------------------------------
This test
* Begins with an inactive follower
* Writes snapshots, each with new keys
* Starts the inactive follower
** The new follower installs a Ratis Snapshot that has all the snapshots and their keys from the leader
* Validates the follower’s data from the latest snapshot
During the data validation, the test checks that every snapshot file that is a hard-link on the leader is also a hard-link on the follower.
When the test is run repeatedly on a remote workflow, it fails almost 50% of the time. As a result, the test has been marked as unhealthy and removed from the CI. During these failures, 1 or more files that can be found on the leader's active fs and are hard-links, don’t exist under the follower’s active fs.
*The test assumes that if the file exists on both directories, active fs and snapshot, then there is a hard link. The follower gets a fraction of the sst files that exist on the leader’s active fs dir and that could be the issue.*
To elaborate, this is how the follower gets the files found on the active fs dir
# The leader gets a RocksDB checkpoint
# The files from the checkpoint get stored under a new dir *leader_node/db.checkpoints/om.db_checkpoint_…*
# The checkpoint files will be stored on the root of tarball and will end up under the active fs dir on the follower
# A hardLinkFile is created that contains all the hard links that the follower must create between the files in the active fs and the snapshot directories
I have gone through the entire process from the leader getting the checkpoint up until the follower installing the tarball and creating the hard links. The missing files aren’t part of the RocksDB checkpoint. Therefore, we can’t find them in the checkpoint dir, the tarball root and the hardLinkFile. The hardLinkFile contains references to the snapshots that contain these sst files but not the active fs root.
The RocksDB checkpoint contains only a small part of the files existing in the leader’s active fs dir. This is an example of the number of files found in both dirs
*leader.activeFs number of files: 264*
*leader.snapshot number of files: 52*
*follower.activeFs number of files: 59*
*follower.snapshot number of files: 58*
The test does the following
# Gets all the files in the leader's snapshot dir
# For every file,
## Gets the filename
## If it exists under the leader's active fs
## Checks if the file on the active fs and the file on the snapshot dir, on the leader, are hard-links
## If they are hard-links on the leader, asserts that they are hard-links on the follower as well
As mentioned above, not all sst files that can be found on the leader's active fs, are included in a checkpoint. Because we are iterating the leader's sst files, there is a very high chance that these files won't be present on the follower.
The above steps can be changed to
# Get all the files from the follower's snapshot dir
# For every file
## Get the filename
## If it exists on the follower's active fs AND if it exists on the leader's active fs
## Check whether the snapshot and active fs files are hard-links on the leader
## If they are hard-links on the leader, assert that they are hard-links on the follower as well
The above approach makes the test pass 10x10 : [https://github.com/xBis7/ozone/actions/runs/7507693387]
[~hemantk] Does this seem like a reasonable change? Or do you think there is a bug here that needs further investigation?
The previous approach was checking that all hard-links on the leader, are present on the follower. This approach checks only the files that can be found on the follower. If these files are hard-links on the leader, then check that these files are hard-links on the follower as well.
was (Author: JIRAUSER285705):
This test
* Begins with an inactive follower
* Writes snapshots, each with new keys
* Starts the inactive follower
** The new follower installs a Ratis Snapshot that has all the snapshots and their keys from the leader
* Validates the follower’s data from the latest snapshot
During the data validation, the test checks that every snapshot file that is a hard-link on the leader is also a hard-link on the follower.
When the test is run repeatedly on a remote workflow, it fails almost 50% of the time. As a result, the test has been marked as unhealthy and removed from the CI. During these failures, 1 or more files that can be found on the leader's active fs and are hard-links, don’t exist under the follower’s active fs.
*The test assumes that if the file exists on both directories, active fs and snapshot, then there is a hard link. The follower gets a fraction of the sst files that exist on the leader’s active fs dir and that could be the issue.*
To elaborate, this is how the follower gets the files found on the active fs dir
# The leader gets a RocksDB checkpoint
# The files from the checkpoint get stored under a new dir *leader_node/db.checkpoints/om.db_checkpoint_…*
# The checkpoint files will be stored on the root of tarball and will end up under the active fs dir on the follower
# A hardLinkFile is created that contains all the hard links that the follower must create between the files in the active fs and the snapshot directories
I have gone through the entire process from the leader getting the checkpoint up until the follower installing the tarball and creating the hard links. The missing files aren’t part of the RocksDB checkpoint. Therefore, we can’t find them in the checkpoint dir, the tarball root and the hardLinkFile. The hardLinkFile contains references to the snapshots that contain these sst files but not the active fs root.
The RocksDB checkpoint contains only a small part of the files existing in the leader’s active fs dir. This is an example of the number of files found in both dirs
*leader.activeFs number of files: 264*
*leader.snapshot number of files: 52*
*follower.activeFs number of files: 59*
*follower.snapshot number of files: 58*
The test does the following
# Gets all the files in the leader's snapshot dir
# For every file,
## Gets the filename
## If it exists under the leader's active fs
## Checks if the file on the active fs and the file on the snapshot dir, on the leader, are hard-links
## If they are hard-links on the leader, asserts that they are hard-links on the follower as well
As mentioned above, not all sst files that can be found on the leader's active fs, are included in a checkpoint. Because we are iterating the leader's sst files, there is a very high chance that these files won't be present on the follower.
The above steps can be changed to
# Get all the files from the follower's snapshot dir
# For every file
## Get the filename
## If it exists on the follower's active fs AND if it exists on the leader's active fs
## Check whether the snapshot and active fs files are hard-links on the leader
## If they are hard-links on the leader, assert that they are hard-links on the follower as well
The above approach makes the test pass 10x10 : [https://github.com/xBis7/ozone/actions/runs/7507693387|https://github.com/xBis7/ozone/actions/runs/7507693387]
[~hemantk] Does this seem like a reasonable change? Or do you think there is a bug here that needs further investigation?
> [disabled] Intermittent failure in TestOMRatisSnapshots.testInstallSnapshot
> ---------------------------------------------------------------------------
>
> Key: HDDS-10059
> URL: https://issues.apache.org/jira/browse/HDDS-10059
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: test
> Reporter: Attila Doroszlai
> Assignee: Christos Bisias
> Priority: Major
>
> Failure 1:
> {code:title=https://github.com/adoroszlai/ozone-build-results/blob/master/2023/12/30/27977/it-om/hadoop-ozone/integration-test/org.apache.hadoop.ozone.om.TestOMRatisSnapshots.txt}
> org.apache.hadoop.ozone.om.TestOMRatisSnapshots.testInstallSnapshot(int, Path)[1] -- Time elapsed: 90.79 s <<< ERROR!
> java.io.IOException: snapshot directory doesn't exist
> at org.apache.hadoop.ozone.om.TestOMRatisSnapshots.createOzoneSnapshot(TestOMRatisSnapshots.java:1060)
> at org.apache.hadoop.ozone.om.TestOMRatisSnapshots.testInstallSnapshot(TestOMRatisSnapshots.java:238)
> {code}
> Failure 2:
> {code:title=https://github.com/adoroszlai/ozone-build-results/blob/master/2024/01/03/28076/it-om/hadoop-ozone/integration-test/org.apache.hadoop.ozone.om.TestOMRatisSnapshots.txt}
> org.apache.hadoop.ozone.om.TestOMRatisSnapshots.testInstallSnapshot(int, Path)[1] -- Time elapsed: 85.34 s <<< ERROR!
> java.nio.file.NoSuchFileException: /home/runner/work/ozone/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-3a4cbda0-c8c0-415c-b2a8-04058ca404e1/omNode-3/om.db/000786.sst
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
> at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
> at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
> at java.nio.file.Files.readAttributes(Files.java:1737)
> at org.apache.hadoop.ozone.om.snapshot.OmSnapshotUtils.getINode(OmSnapshotUtils.java:67)
> at org.apache.hadoop.ozone.om.TestOMRatisSnapshots.checkSnapshot(TestOMRatisSnapshots.java:373)
> at org.apache.hadoop.ozone.om.TestOMRatisSnapshots.testInstallSnapshot(TestOMRatisSnapshots.java:312)
> {code}
> [~xBis] would you like to take a look?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org