[GitHub] [bookkeeper] hangc0276 commented on a diff in pull request #3924: [stats][bookie] Add bookie sanity test metrics

["hangc0276 (via GitHub)" <gi...@apache.org>]

hangc0276 commented on code in PR #3924:
URL: https://github.com/apache/bookkeeper/pull/3924#discussion_r1169448534


##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieStateManager.java:
##########
@@ -149,6 +160,30 @@ public Number getSample() {
             }
         };
         statsLogger.registerGauge(SERVER_STATUS, serverStatusGauge);
+        this.serverSanityGauge = new Gauge<Number>() {
+            @Override
+            public Number getDefaultValue() {
+                return -1;
+            }
+
+            @Override
+            public Number getSample() {
+                return sanityPassed.get();
+            }
+        };
+        statsLogger.registerGauge(SERVER_SANITY, serverSanityGauge);
+        stateService.scheduleAtFixedRate(() -> {
+            if (forceReadOnly.get()) {
+                sanityPassed.set(1);
+                return;
+            }
+            SanityTestCommand.handleAsync(conf, new SanityTestCommand.SanityFlags()).thenAccept(__ -> {

Review Comment:
   Do we need to check whether the bookie is in ReadOnly mode? so that we can skip running the Sanity test.



##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/tools/cli/commands/bookie/SanityTestCommand.java:
##########
@@ -81,59 +86,126 @@ public boolean apply(ServerConfiguration conf, SanityFlags cmdFlags) {
         }
     }
 
-    private boolean handle(ServerConfiguration conf, SanityFlags cmdFlags) throws Exception {
+    private static boolean handle(ServerConfiguration conf, SanityFlags cmdFlags) throws Exception {
+        try {
+            return handleAsync(conf, cmdFlags).get();
+        } catch (Exception e) {
+            LOG.warn("Error in bookie sanity test", e);
+            return false;
+        }
+    }
+
+    public static CompletableFuture<Boolean> handleAsync(ServerConfiguration conf, SanityFlags cmdFlags) {
+        CompletableFuture<Boolean> result = new CompletableFuture<Boolean>();
         ClientConfiguration clientConf = new ClientConfiguration();
         clientConf.addConfiguration(conf);
         clientConf.setEnsemblePlacementPolicy(LocalBookieEnsemblePlacementPolicy.class);
         clientConf.setAddEntryTimeout(cmdFlags.timeout);
         clientConf.setReadEntryTimeout(cmdFlags.timeout);
 
-        BookKeeper bk = new BookKeeper(clientConf);
-        LedgerHandle lh = null;
+        BookKeeper bk;
         try {
-            lh = bk.createLedger(1, 1, BookKeeper.DigestType.MAC, new byte[0]);
-            LOG.info("Create ledger {}", lh.getId());
+            bk = new BookKeeper(clientConf);
+        } catch (BKException | IOException | InterruptedException e) {
+            LOG.warn("Failed to initialize bookkeeper client", e);
+            result.completeExceptionally(e);
+            return result;
+        }
 
+        bk.asyncCreateLedger(1, 1, BookKeeper.DigestType.MAC, new byte[0], (rc, lh, ctx) -> {
+            if (rc != BKException.Code.OK) {
+                LOG.warn("ledger creation failed for sanity command {}", rc);
+                result.completeExceptionally(BKException.create(rc));
+                return;
+            }
+            List<CompletableFuture<Void>> entriesFutures = new ArrayList<>();
             for (int i = 0; i < cmdFlags.entries; i++) {
                 String content = "entry-" + i;
-                lh.addEntry(content.getBytes(UTF_8));
-            }
-
-            LOG.info("Written {} entries in ledger {}", cmdFlags.entries, lh.getId());
-
-            // Reopen the ledger and read entries
-            lh = bk.openLedger(lh.getId(), BookKeeper.DigestType.MAC, new byte[0]);
-            if (lh.getLastAddConfirmed() != (cmdFlags.entries - 1)) {
-                throw new Exception("Invalid last entry found on ledger. expecting: " + (cmdFlags.entries - 1)
-                                        + " -- found: " + lh.getLastAddConfirmed());
+                CompletableFuture<Void> entryFuture = new CompletableFuture<>();
+                entriesFutures.add(entryFuture);
+                lh.asyncAddEntry(content.getBytes(UTF_8), (arc, alh, entryId, actx) -> {
+                    if (arc != BKException.Code.OK) {
+                        LOG.warn("ledger add entry failed for {}-{}", alh.getId(), arc);
+                        entryFuture.completeExceptionally(BKException.create(arc));
+                        return;
+                    }
+                    entryFuture.complete(null);
+                }, null);
             }
+            CompletableFuture<LedgerHandle> lhFuture = new CompletableFuture<>();
+            CompletableFuture<Void> readEntryFuture = new CompletableFuture<>();
+
+            FutureUtils.collect(entriesFutures).thenCompose(_r -> {
+                bk.asyncOpenLedger(lh.getId(), BookKeeper.DigestType.MAC, new byte[0], (orc, olh, octx) -> {
+                    if (orc != BKException.Code.OK) {
+                        LOG.warn("open sanity ledger failed for {}-{}", lh.getId(), orc);
+                        lhFuture.completeExceptionally(BKException.create(orc));
+                        return;
+                    }
+                    long lac = olh.getLastAddConfirmed();
+                    if (lac != (cmdFlags.entries - 1)) {
+                        lhFuture.completeExceptionally(new Exception("Invalid last entry found on ledger. expecting: "
+                                + (cmdFlags.entries - 1) + " -- found: " + lac));
+                        return;
+                    }
+                    lhFuture.complete(lh);
+                }, null);

Review Comment:
   We'd better add the `customMetadata` for the created ledger in order to find out orphan ledgers if the ledger delete failed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org