[GitHub] [accumulo] milleruntime commented on a change in pull request #2583: Add trace and debug log to consistency check

[GitBox <gi...@apache.org>]

milleruntime commented on a change in pull request #2583:
URL: https://github.com/apache/accumulo/pull/2583#discussion_r833116378



##########
File path: server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
##########
@@ -809,27 +814,40 @@ public void run() {
     // Periodically check that metadata of tablets matches what is held in memory
     ThreadPools.watchCriticalScheduledTask(ThreadPools.getServerThreadPools()
         .createGeneralScheduledExecutorService(aconf).scheduleWithFixedDelay(() -> {
-          final SortedMap<KeyExtent,Tablet> onlineTabletsSnapshot = onlineTablets.snapshot();
-
-          Map<KeyExtent,Long> updateCounts = new HashMap<>();
+          Instant start = Instant.now();
+          Span mdScanSpan = TraceUtil.startSpan(this.getClass(), "metadataScan");
+          try (Scope scope = mdScanSpan.makeCurrent()) {
+            final SortedMap<KeyExtent,Tablet> onlineTabletsSnapshot = onlineTablets.snapshot();
 
-          // gather updateCounts for each tablet
-          onlineTabletsSnapshot.forEach((ke, tablet) -> {
-            updateCounts.put(ke, tablet.getUpdateCount());
-          });
+            Map<KeyExtent,Long> updateCounts = new HashMap<>();
 
-          // gather metadata for all tablets readTablets()
-          try (TabletsMetadata tabletsMetadata =
-              getContext().getAmple().readTablets().forTablets(onlineTabletsSnapshot.keySet())
-                  .fetch(FILES, LOGS, ECOMP, PREV_ROW).build()) {
-
-            // for each tablet, compare its metadata to what is held in memory
-            tabletsMetadata.forEach(tabletMetadata -> {
-              KeyExtent extent = tabletMetadata.getExtent();
-              Tablet tablet = onlineTabletsSnapshot.get(extent);
-              Long counter = updateCounts.get(extent);
-              tablet.compareTabletInfo(counter, tabletMetadata);
+            // gather updateCounts for each tablet
+            onlineTabletsSnapshot.forEach((ke, tablet) -> {
+              updateCounts.put(ke, tablet.getUpdateCount());
             });
+
+            // gather metadata for all tablets readTablets()
+            try (TabletsMetadata tabletsMetadata =
+                getContext().getAmple().readTablets().forTablets(onlineTabletsSnapshot.keySet())
+                    .fetch(FILES, LOGS, ECOMP, PREV_ROW).build()) {
+
+              // for each tablet, compare its metadata to what is held in memory
+              tabletsMetadata.forEach(tabletMetadata -> {
+                KeyExtent extent = tabletMetadata.getExtent();
+                Tablet tablet = onlineTabletsSnapshot.get(extent);
+                Long counter = updateCounts.get(extent);
+                tablet.compareTabletInfo(counter, tabletMetadata);
+              });
+            }
+          } finally {
+            mdScanSpan.end();
+            Instant end = Instant.now();
+            Duration duration = Duration.between(start, end);
+            log.debug("Metadata scan took {}ms", duration.toMillis());
+            if (duration.toMinutes() > 5) {
+              log.warn(
+                  "Metadata scan is taking too long. Performance of all activities will be severely hindered!");

Review comment:
       Ah yeah, good idea.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org