You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "devmadhuu (via GitHub)" <gi...@apache.org> on 2026/05/05 08:31:59 UTC
[PR] HDDS-15165. Recon: Add admin REST APIs to trigger, monitor, and cancel SCM DB snapshot sync. [ozone]
devmadhuu opened a new pull request, #10186:
URL: https://github.com/apache/ozone/pull/10186
## What changes were proposed in this pull request?
This change adds Recon admin REST APIs to explicitly manage full SCM DB snapshot syncs as a follow-up to HDDS-14758.
The new APIs are exposed under `TriggerDBSyncEndpoint`:
```
- POST /api/v1/triggerdbsync/scm/snapshot triggers an async full SCM DB snapshot download and refresh.
- GET /api/v1/triggerdbsync/scm/snapshot/status returns current status, phase, start time, finish time, duration, cancellation eligibility, and last error.
- POST /api/v1/triggerdbsync/scm/snapshot/cancel cancels the operation while checkpoint download is still in progress.
```
The implementation keeps full SCM snapshot recovery as an explicit admin action instead of coupling it to periodic SCM container sync. It tracks snapshot sync lifecycle state in `ReconStorageContainerManagerFacade`, prevents concurrent SCM DB sync operations, allows cancellation only before DB initialization starts, and cleans up failed or cancelled checkpoint directories.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15165
## How was this patch tested?
Tests were added for the new trigger/status/cancel endpoint behavior in `TestTriggerDBSyncEndpoint`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15165. Recon: Add admin REST APIs to trigger, monitor, and cancel SCM DB snapshot sync. [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #10186:
URL: https://github.com/apache/ozone/pull/10186#discussion_r3242023499
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -607,19 +921,15 @@ private void initializeNewRdbStore(File dbFile) throws IOException {
ReconSCMDBDefinition.CONTAINERS.getTable(newStore));
nodeManager.reinitialize(
ReconSCMDBDefinition.NODES.getTable(newStore));
- IOUtils.close(LOG, dbStore);
- deleteOldSCMDB();
dbStore = newStore;
- File newDb = new File(dbFile.getParent() +
- OZONE_URI_DELIMITER + ReconSCMDBDefinition.RECON_SCM_DB_NAME);
- boolean success = dbFile.renameTo(newDb);
Review Comment:
Since rename is done earlier, but now its removed. Do this have any impact ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15165. Recon: Add admin REST APIs to trigger, monitor, and cancel SCM DB snapshot sync. [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #10186:
URL: https://github.com/apache/ozone/pull/10186#discussion_r3196228970
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -550,23 +711,162 @@ private void initializeSCMDB() {
public void updateReconSCMDBWithNewSnapshot() throws IOException {
if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
- try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
- } catch (IOException e) {
- LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
- }
- } else {
- LOG.error("Null snapshot location got from SCM.");
+ try {
+ updateReconSCMDBWithNewSnapshotWithoutGuard();
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
} else {
LOG.warn("SCM DB sync is already running.");
}
}
+ private void updateReconSCMDBWithNewSnapshotWithoutGuard()
+ throws IOException {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : {}",
+ dbSnapshot.getCheckpointLocation());
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } else {
+ throw new IOException("Null snapshot location got from SCM.");
+ }
+ }
+
+ public ScmDbSnapshotTriggerResponse triggerScmDbSnapshotSync() {
+ synchronized (scmSnapshotLock) {
+ if (!isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ return new ScmDbSnapshotTriggerResponse(false, scmSnapshotStatus,
+ "SCM DB sync is already running.");
+ }
+ scmSnapshotStatus = ScmDbSnapshotSyncStatus.IN_PROGRESS;
+ scmSnapshotPhase = ScmDbSnapshotSyncPhase.DOWNLOADING_CHECKPOINT;
+ scmSnapshotStartedAt = System.currentTimeMillis();
+ scmSnapshotFinishedAt = 0;
+ scmSnapshotCancelAllowed = true;
+ scmSnapshotTaskStarted = false;
+ scmSnapshotLastError = null;
+ scmSnapshotFuture = scmSnapshotExecutor.submit(this::runScmSnapshotSync);
+ return new ScmDbSnapshotTriggerResponse(true, scmSnapshotStatus,
+ "SCM DB snapshot sync started.");
+ }
+ }
+
+ public ScmDbSnapshotStatusResponse getScmDbSnapshotSyncStatus() {
+ synchronized (scmSnapshotLock) {
+ return new ScmDbSnapshotStatusResponse(scmSnapshotStatus,
+ scmSnapshotPhase, scmSnapshotStartedAt, scmSnapshotFinishedAt,
+ scmSnapshotCancelAllowed, scmSnapshotLastError);
+ }
+ }
+
+ public ScmDbSnapshotCancelResponse cancelScmDbSnapshotSync() {
+ synchronized (scmSnapshotLock) {
+ if (scmSnapshotStatus != ScmDbSnapshotSyncStatus.IN_PROGRESS) {
+ return new ScmDbSnapshotCancelResponse(false, scmSnapshotStatus,
+ scmSnapshotPhase, "No SCM DB snapshot sync is running.");
+ }
+ if (!scmSnapshotCancelAllowed) {
+ return new ScmDbSnapshotCancelResponse(false, scmSnapshotStatus,
+ scmSnapshotPhase,
+ "Cancellation is not allowed after DB initialization has started.");
+ }
+ boolean cancelled = scmSnapshotFuture != null &&
+ scmSnapshotFuture.cancel(true);
+ if (cancelled) {
+ scmSnapshotStatus = ScmDbSnapshotSyncStatus.CANCELLED;
+ scmSnapshotPhase = ScmDbSnapshotSyncPhase.CANCELLED;
+ scmSnapshotFinishedAt = System.currentTimeMillis();
+ scmSnapshotCancelAllowed = false;
+ if (!scmSnapshotTaskStarted) {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
+ }
+ }
+ return new ScmDbSnapshotCancelResponse(cancelled, scmSnapshotStatus,
+ scmSnapshotPhase, cancelled ? "SCM DB snapshot sync cancelled." :
+ "Unable to cancel SCM DB snapshot sync.");
+ }
+ }
+
+ private void runScmSnapshotSync() {
+ File checkpointLocation = null;
+ boolean initialized = false;
+ try {
+ synchronized (scmSnapshotLock) {
+ scmSnapshotTaskStarted = true;
+ }
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
Review Comment:
a check is required here if cancelled or not, as there is a chance of cancel marked cancel, and set isSyncDataFromSCMRunning to false. But its still running waiting in synchronized block. This check should be added in synchronized block
This needs to be done before starting sync db from scm
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -550,23 +711,162 @@ private void initializeSCMDB() {
public void updateReconSCMDBWithNewSnapshot() throws IOException {
if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
- try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
- } catch (IOException e) {
- LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
- }
- } else {
- LOG.error("Null snapshot location got from SCM.");
+ try {
+ updateReconSCMDBWithNewSnapshotWithoutGuard();
Review Comment:
do startup sync and trigger can have some conflict ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15165. Recon: Add admin REST APIs to trigger, monitor, and cancel SCM DB snapshot sync. [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl merged PR #10186:
URL: https://github.com/apache/ozone/pull/10186
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15165. Recon: Add admin REST APIs to trigger, monitor, and cancel SCM DB snapshot sync. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #10186:
URL: https://github.com/apache/ozone/pull/10186#discussion_r3200612620
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -550,23 +711,162 @@ private void initializeSCMDB() {
public void updateReconSCMDBWithNewSnapshot() throws IOException {
if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
- try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
- } catch (IOException e) {
- LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
- }
- } else {
- LOG.error("Null snapshot location got from SCM.");
+ try {
+ updateReconSCMDBWithNewSnapshotWithoutGuard();
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
} else {
LOG.warn("SCM DB sync is already running.");
}
}
+ private void updateReconSCMDBWithNewSnapshotWithoutGuard()
+ throws IOException {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : {}",
+ dbSnapshot.getCheckpointLocation());
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } else {
+ throw new IOException("Null snapshot location got from SCM.");
+ }
+ }
+
+ public ScmDbSnapshotTriggerResponse triggerScmDbSnapshotSync() {
+ synchronized (scmSnapshotLock) {
+ if (!isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ return new ScmDbSnapshotTriggerResponse(false, scmSnapshotStatus,
+ "SCM DB sync is already running.");
+ }
+ scmSnapshotStatus = ScmDbSnapshotSyncStatus.IN_PROGRESS;
+ scmSnapshotPhase = ScmDbSnapshotSyncPhase.DOWNLOADING_CHECKPOINT;
+ scmSnapshotStartedAt = System.currentTimeMillis();
+ scmSnapshotFinishedAt = 0;
+ scmSnapshotCancelAllowed = true;
+ scmSnapshotTaskStarted = false;
+ scmSnapshotLastError = null;
+ scmSnapshotFuture = scmSnapshotExecutor.submit(this::runScmSnapshotSync);
+ return new ScmDbSnapshotTriggerResponse(true, scmSnapshotStatus,
+ "SCM DB snapshot sync started.");
+ }
+ }
+
+ public ScmDbSnapshotStatusResponse getScmDbSnapshotSyncStatus() {
+ synchronized (scmSnapshotLock) {
+ return new ScmDbSnapshotStatusResponse(scmSnapshotStatus,
+ scmSnapshotPhase, scmSnapshotStartedAt, scmSnapshotFinishedAt,
+ scmSnapshotCancelAllowed, scmSnapshotLastError);
+ }
+ }
+
+ public ScmDbSnapshotCancelResponse cancelScmDbSnapshotSync() {
+ synchronized (scmSnapshotLock) {
+ if (scmSnapshotStatus != ScmDbSnapshotSyncStatus.IN_PROGRESS) {
+ return new ScmDbSnapshotCancelResponse(false, scmSnapshotStatus,
+ scmSnapshotPhase, "No SCM DB snapshot sync is running.");
+ }
+ if (!scmSnapshotCancelAllowed) {
+ return new ScmDbSnapshotCancelResponse(false, scmSnapshotStatus,
+ scmSnapshotPhase,
+ "Cancellation is not allowed after DB initialization has started.");
+ }
+ boolean cancelled = scmSnapshotFuture != null &&
+ scmSnapshotFuture.cancel(true);
+ if (cancelled) {
+ scmSnapshotStatus = ScmDbSnapshotSyncStatus.CANCELLED;
+ scmSnapshotPhase = ScmDbSnapshotSyncPhase.CANCELLED;
+ scmSnapshotFinishedAt = System.currentTimeMillis();
+ scmSnapshotCancelAllowed = false;
+ if (!scmSnapshotTaskStarted) {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
+ }
+ }
+ return new ScmDbSnapshotCancelResponse(cancelled, scmSnapshotStatus,
+ scmSnapshotPhase, cancelled ? "SCM DB snapshot sync cancelled." :
+ "Unable to cancel SCM DB snapshot sync.");
+ }
+ }
+
+ private void runScmSnapshotSync() {
+ File checkpointLocation = null;
+ boolean initialized = false;
+ try {
+ synchronized (scmSnapshotLock) {
+ scmSnapshotTaskStarted = true;
+ }
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
Review Comment:
Ah, thanks for catching this race condition. Yes I fixed it now.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -550,23 +711,162 @@ private void initializeSCMDB() {
public void updateReconSCMDBWithNewSnapshot() throws IOException {
if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
- try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
- } catch (IOException e) {
- LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
- }
- } else {
- LOG.error("Null snapshot location got from SCM.");
+ try {
+ updateReconSCMDBWithNewSnapshotWithoutGuard();
Review Comment:
Yes . fixed it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org