You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "aswinshakil (via GitHub)" <gi...@apache.org> on 2023/06/21 17:55:18 UTC
[GitHub] [ozone] aswinshakil opened a new pull request, #4946: HDDS-8685. [SNAPSHOT] CreateCheckpoint needs to be idempotent.
aswinshakil opened a new pull request, #4946:
URL: https://github.com/apache/ozone/pull/4946
## What changes were proposed in this pull request?
Checkpoint creation should be idempotent in the case of log replays.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8685
## How was this patch tested?
Existing UT and New UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] prashantpogde commented on a diff in pull request #4946: HDDS-8685. [SNAPSHOT] CreateCheckpoint needs to be idempotent.
Posted by "prashantpogde (via GitHub)" <gi...@apache.org>.
prashantpogde commented on code in PR #4946:
URL: https://github.com/apache/ozone/pull/4946#discussion_r1237600999
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -386,16 +389,30 @@ public static DBCheckpoint createOmSnapshotCheckpoint(
omMetadataManager.getTableLock(OmMetadataManagerImpl.DELETED_TABLE)
.writeLock().lock();
+ boolean snapshotDirExist = false;
+
try {
// Create DB checkpoint for snapshot
- dbCheckpoint = store.getSnapshot(snapshotInfo.getCheckpointDirName());
- // Clean up active DB's deletedTable right after checkpoint is taken,
- // with table write lock held
- deleteKeysFromDelKeyTableInSnapshotScope(omMetadataManager,
- snapshotInfo.getVolumeName(), snapshotInfo.getBucketName());
- // Clean up deletedDirectoryTable as well
- deleteKeysFromDelDirTableInSnapshotScope(omMetadataManager,
- snapshotInfo.getVolumeName(), snapshotInfo.getBucketName());
+ String checkpointPrefix = store.getDbLocation().getName();
+ Path snapshotDirPath = Paths.get(store.getSnapshotsParentDir(),
+ checkpointPrefix + snapshotInfo.getCheckpointDir());
+ if (Files.exists(snapshotDirPath)) {
+ snapshotDirExist = true;
+ dbCheckpoint = new RocksDBCheckpoint(snapshotDirPath);
+ } else {
+ dbCheckpoint = store.getSnapshot(snapshotInfo.getCheckpointDirName());
Review Comment:
In this path we setup checkpointCreationTimeTaken etc. on the 3 OM HA nodes, the creation time could differ, this should have come in the request. Can you check the other path ? cc : @hemantk-12
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] prashantpogde commented on a diff in pull request #4946: HDDS-8685. [SNAPSHOT] CreateCheckpoint needs to be idempotent.
Posted by "prashantpogde (via GitHub)" <gi...@apache.org>.
prashantpogde commented on code in PR #4946:
URL: https://github.com/apache/ozone/pull/4946#discussion_r1237757643
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -386,16 +389,30 @@ public static DBCheckpoint createOmSnapshotCheckpoint(
omMetadataManager.getTableLock(OmMetadataManagerImpl.DELETED_TABLE)
.writeLock().lock();
+ boolean snapshotDirExist = false;
+
try {
// Create DB checkpoint for snapshot
- dbCheckpoint = store.getSnapshot(snapshotInfo.getCheckpointDirName());
- // Clean up active DB's deletedTable right after checkpoint is taken,
- // with table write lock held
- deleteKeysFromDelKeyTableInSnapshotScope(omMetadataManager,
- snapshotInfo.getVolumeName(), snapshotInfo.getBucketName());
- // Clean up deletedDirectoryTable as well
- deleteKeysFromDelDirTableInSnapshotScope(omMetadataManager,
- snapshotInfo.getVolumeName(), snapshotInfo.getBucketName());
+ String checkpointPrefix = store.getDbLocation().getName();
+ Path snapshotDirPath = Paths.get(store.getSnapshotsParentDir(),
+ checkpointPrefix + snapshotInfo.getCheckpointDir());
+ if (Files.exists(snapshotDirPath)) {
+ snapshotDirExist = true;
+ dbCheckpoint = new RocksDBCheckpoint(snapshotDirPath);
+ } else {
+ dbCheckpoint = store.getSnapshot(snapshotInfo.getCheckpointDirName());
+ }
+
+ if (!snapshotDirExist) {
Review Comment:
Yes, we should always clean this up. We don't know when the system crash occured.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on pull request #4946: HDDS-8685. [SNAPSHOT] CreateCheckpoint needs to be idempotent.
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #4946:
URL: https://github.com/apache/ozone/pull/4946#issuecomment-1601823034
findbugs in test:
```
M D DLS: Dead store to omSnapshotCheckpoint in org.apache.hadoop.ozone.om.TestOmSnapshotManager.testCreateSnapshotIdempotent() At TestOmSnapshotManager.java:[line 428]
M D DLS: Dead store to omSnapshotCheckpoint1 in org.apache.hadoop.ozone.om.TestOmSnapshotManager.testCreateSnapshotIdempotent() At TestOmSnapshotManager.java:[line 436]
```
https://github.com/apache/ozone/actions/runs/5337343478/jobs/9673267601
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] aswinshakil commented on a diff in pull request #4946: HDDS-8685. [SNAPSHOT] CreateCheckpoint needs to be idempotent.
Posted by "aswinshakil (via GitHub)" <gi...@apache.org>.
aswinshakil commented on code in PR #4946:
URL: https://github.com/apache/ozone/pull/4946#discussion_r1237381573
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -386,16 +389,30 @@ public static DBCheckpoint createOmSnapshotCheckpoint(
omMetadataManager.getTableLock(OmMetadataManagerImpl.DELETED_TABLE)
.writeLock().lock();
+ boolean snapshotDirExist = false;
+
try {
// Create DB checkpoint for snapshot
- dbCheckpoint = store.getSnapshot(snapshotInfo.getCheckpointDirName());
- // Clean up active DB's deletedTable right after checkpoint is taken,
- // with table write lock held
- deleteKeysFromDelKeyTableInSnapshotScope(omMetadataManager,
- snapshotInfo.getVolumeName(), snapshotInfo.getBucketName());
- // Clean up deletedDirectoryTable as well
- deleteKeysFromDelDirTableInSnapshotScope(omMetadataManager,
- snapshotInfo.getVolumeName(), snapshotInfo.getBucketName());
+ String checkpointPrefix = store.getDbLocation().getName();
+ Path snapshotDirPath = Paths.get(store.getSnapshotsParentDir(),
+ checkpointPrefix + snapshotInfo.getCheckpointDir());
+ if (Files.exists(snapshotDirPath)) {
+ snapshotDirExist = true;
+ dbCheckpoint = new RocksDBCheckpoint(snapshotDirPath);
+ } else {
+ dbCheckpoint = store.getSnapshot(snapshotInfo.getCheckpointDirName());
+ }
+
+ if (!snapshotDirExist) {
Review Comment:
Should we do the cleanup even if the checkpoint already exists?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] prashantpogde commented on a diff in pull request #4946: HDDS-8685. [SNAPSHOT] CreateCheckpoint needs to be idempotent.
Posted by "prashantpogde (via GitHub)" <gi...@apache.org>.
prashantpogde commented on code in PR #4946:
URL: https://github.com/apache/ozone/pull/4946#discussion_r1237600999
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -386,16 +389,30 @@ public static DBCheckpoint createOmSnapshotCheckpoint(
omMetadataManager.getTableLock(OmMetadataManagerImpl.DELETED_TABLE)
.writeLock().lock();
+ boolean snapshotDirExist = false;
+
try {
// Create DB checkpoint for snapshot
- dbCheckpoint = store.getSnapshot(snapshotInfo.getCheckpointDirName());
- // Clean up active DB's deletedTable right after checkpoint is taken,
- // with table write lock held
- deleteKeysFromDelKeyTableInSnapshotScope(omMetadataManager,
- snapshotInfo.getVolumeName(), snapshotInfo.getBucketName());
- // Clean up deletedDirectoryTable as well
- deleteKeysFromDelDirTableInSnapshotScope(omMetadataManager,
- snapshotInfo.getVolumeName(), snapshotInfo.getBucketName());
+ String checkpointPrefix = store.getDbLocation().getName();
+ Path snapshotDirPath = Paths.get(store.getSnapshotsParentDir(),
+ checkpointPrefix + snapshotInfo.getCheckpointDir());
+ if (Files.exists(snapshotDirPath)) {
+ snapshotDirExist = true;
+ dbCheckpoint = new RocksDBCheckpoint(snapshotDirPath);
+ } else {
+ dbCheckpoint = store.getSnapshot(snapshotInfo.getCheckpointDirName());
Review Comment:
In this path we setup checkpointCreationTimeTaken etc. on the 3 OM HA nodes, the creation time could differ, this should have come in the request. Can you check the other path ? cc : @hemantk-12
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] prashantpogde merged pull request #4946: HDDS-8685. [SNAPSHOT] CreateCheckpoint needs to be idempotent.
Posted by "prashantpogde (via GitHub)" <gi...@apache.org>.
prashantpogde merged PR #4946:
URL: https://github.com/apache/ozone/pull/4946
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org