You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "dlmarion (via GitHub)" <gi...@apache.org> on 2024/07/24 11:53:48 UTC

Re: [PR] Cleans up non-existent migrating metadata tablets in manager [accumulo]

dlmarion commented on code in PR #4750:
URL: https://github.com/apache/accumulo/pull/4750#discussion_r1689637175


##########
server/manager/src/main/java/org/apache/accumulo/manager/Manager.java:
##########
@@ -712,21 +708,35 @@ public void run() {
      * migration will refer to a non-existing tablet, so it can never complete. Periodically scan
      * the metadata table and remove any migrating tablets that no longer exist.
      */
-    private void cleanupNonexistentMigrations(final AccumuloClient accumuloClient)
+    private void cleanupNonexistentMigrations(final ClientContext clientContext)
         throws TableNotFoundException {
-      Scanner scanner = accumuloClient.createScanner(MetadataTable.NAME, Authorizations.EMPTY);
-      TabletColumnFamily.PREV_ROW_COLUMN.fetch(scanner);
-      scanner.setRange(MetadataSchema.TabletsSection.getRange());
-      Set<KeyExtent> notSeen;
+
+      Map<DataLevel,Set<KeyExtent>> notSeen = new EnumMap<>(DataLevel.class);
+
       synchronized (migrations) {
-        notSeen = new HashSet<>(migrations.keySet());
+        // partition migrations by level
+        migrations.keySet().forEach(extent -> notSeen
+            .computeIfAbsent(DataLevel.of(extent.tableId()), k -> new HashSet<>()).add(extent));
       }
-      for (Entry<Key,Value> entry : scanner) {
-        KeyExtent extent = KeyExtent.fromMetaPrevRow(entry);
-        notSeen.remove(extent);
+
+      // for each level find the set of migrating tablets that do not exists in metadata store
+      for (DataLevel dataLevel : DataLevel.values()) {
+        var notSeenForLevel = notSeen.getOrDefault(dataLevel, Set.of());
+        if (notSeenForLevel.isEmpty() || dataLevel == DataLevel.ROOT) {
+          // No need to scan this level if there are no migrations. The root tablet is always
+          // expected to exists, so no need to read its metadata.
+          continue;
+        }
+
+        try (var tablets = clientContext.getAmple().readTablets().forLevel(dataLevel)
+            .fetch(TabletMetadata.ColumnType.PREV_ROW).build()) {
+          tablets.forEach(tabletMeta -> notSeenForLevel.remove(tabletMeta.getExtent()));

Review Comment:
   `notSeenForLevel` will always be the smaller Set when compared to `tablets`, unless all tablets are migrating. Might want to use that as the driver, or maybe use Set.removeAll. Something like:
   
   ```
     HashSet<KeyExtent> tabletExtents = new HashSet<>(tablets.size());
     tablets.forEach(tm -> tabletExtents.add(tm.getExtent());
     notSeenForLevel.removeAll(tabletExtents);
   ```



##########
server/manager/src/main/java/org/apache/accumulo/manager/Manager.java:
##########
@@ -712,21 +708,35 @@ public void run() {
      * migration will refer to a non-existing tablet, so it can never complete. Periodically scan
      * the metadata table and remove any migrating tablets that no longer exist.
      */
-    private void cleanupNonexistentMigrations(final AccumuloClient accumuloClient)
+    private void cleanupNonexistentMigrations(final ClientContext clientContext)
         throws TableNotFoundException {
-      Scanner scanner = accumuloClient.createScanner(MetadataTable.NAME, Authorizations.EMPTY);
-      TabletColumnFamily.PREV_ROW_COLUMN.fetch(scanner);
-      scanner.setRange(MetadataSchema.TabletsSection.getRange());
-      Set<KeyExtent> notSeen;
+
+      Map<DataLevel,Set<KeyExtent>> notSeen = new EnumMap<>(DataLevel.class);
+
       synchronized (migrations) {
-        notSeen = new HashSet<>(migrations.keySet());
+        // partition migrations by level
+        migrations.keySet().forEach(extent -> notSeen

Review Comment:
   A `partitionMigrations` method was merged in yesterday.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org