[GitHub] [hadoop] hanishakoneru commented on a change in pull request #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager

[GitBox <gi...@apache.org>]

hanishakoneru commented on a change in pull request #1259: HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager
URL: https://github.com/apache/hadoop/pull/1259#discussion_r314914818
 
 

 ##########
 File path: hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/OzoneManagerServiceProviderImpl.java
 ##########
 @@ -187,5 +229,119 @@ protected DBCheckpoint getOzoneManagerDBSnapshot() {
     }
     return null;
   }
+
+  /**
+   * Update Local OM DB with new OM DB snapshot.
+   * @throws IOException
+   */
+  @VisibleForTesting
+  void updateReconOmDBWithNewSnapshot() throws IOException {
+    // Obtain the current DB snapshot from OM and
+    // update the in house OM metadata managed DB instance.
+    DBCheckpoint dbSnapshot = getOzoneManagerDBSnapshot();
+    if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+      try {
+        omMetadataManager.updateOmDB(dbSnapshot.getCheckpointLocation()
+            .toFile());
+      } catch (IOException e) {
+        LOG.error("Unable to refresh Recon OM DB Snapshot. ", e);
+      }
+    } else {
+      LOG.error("Null snapshot location got from OM.");
+    }
+  }
+
+  /**
+   * Get Delta updates from OM through RPC call and apply to local OM DB as
+   * well as accumulate in a buffer.
+   * @param fromSequenceNumber from sequence number to request from.
+   * @param omdbUpdatesHandler OM DB updates handler to buffer updates.
+   * @throws IOException when OM RPC request fails.
+   * @throws RocksDBException when writing to RocksDB fails.
+   */
+  @VisibleForTesting
+  void getAndApplyDeltaUpdatesFromOM(
+      long fromSequenceNumber, OMDBUpdatesHandler omdbUpdatesHandler)
+      throws IOException, RocksDBException {
+    DBUpdatesRequest dbUpdatesRequest = DBUpdatesRequest.newBuilder()
+        .setSequenceNumber(fromSequenceNumber).build();
+    DBUpdatesWrapper dbUpdates = ozoneManagerClient.getDBUpdates(
+        dbUpdatesRequest);
+    if (null != dbUpdates) {
+      RDBStore rocksDBStore = (RDBStore)omMetadataManager.getStore();
+      RocksDB rocksDB = rocksDBStore.getDb();
+      LOG.debug("Number of updates received from OM : " +
+          dbUpdates.getData().size());
+      for (byte[] data : dbUpdates.getData()) {
+        WriteBatch writeBatch = new WriteBatch(data);
+        writeBatch.iterate(omdbUpdatesHandler);
+        RDBBatchOperation rdbBatchOperation = new RDBBatchOperation(writeBatch);
+        rdbBatchOperation.commit(rocksDB, new WriteOptions());
+      }
+    }
+  }
+
+  /**
+   * Based on current state of Recon's OM DB, we either get delta updates or
+   * full snapshot from Ozone Manager.
+   */
+  @VisibleForTesting
+  void syncDataFromOM() {
+    long currentSequenceNumber = getCurrentOMDBSequenceNumber();
+    boolean fullSnapshot = false;
+
+    if (currentSequenceNumber <= 0) {
+      fullSnapshot = true;
+    } else {
 
 Review comment:
   Should we check the difference between currentSequenceNum and the latest sequence number of OM DB before deciding whether to get full snapshot or delta updates?
   Lets say currentSequenceNum = 100 and latest sequence number is 1M. Would it not be better to just get the full snapshot and replace the old one? (Not sure if this is applicable to Recon)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org