You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "dlmarion (via GitHub)" <gi...@apache.org> on 2023/10/17 13:55:14 UTC

[PR] Track lastAccessTime in Tablet, not in TabletServer collection [accumulo]

dlmarion opened a new pull request, #3855:
URL: https://github.com/apache/accumulo/pull/3855

   Fixes #3263


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Track lastAccessTime in Tablet, not in TabletServer collection [accumulo]

Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on code in PR #3855:
URL: https://github.com/apache/accumulo/pull/3855#discussion_r1364395295


##########
server/tserver/src/main/java/org/apache/accumulo/tserver/AssignmentHandler.java:
##########
@@ -190,6 +189,7 @@ public void run() {
           server.recentlyUnloadedCache.remove(tablet.getExtent());
         }
       }
+      tablet.setLastAccessTime();

Review Comment:
   Implemented in a7361ea



##########
server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java:
##########
@@ -178,6 +178,8 @@ enum CompactionState {
 
   private final int logId;
 
+  private volatile long lastAccessTime = 0;

Review Comment:
   Implemented in a7361ea



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Track lastAccessTime in Tablet, not in TabletServer collection [accumulo]

Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on code in PR #3855:
URL: https://github.com/apache/accumulo/pull/3855#discussion_r1364396018


##########
server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java:
##########
@@ -1182,41 +1156,27 @@ public void evaluateOnDemandTabletsForUnload() {
 
     final SortedMap<KeyExtent,Tablet> online = getOnlineTablets();
 
-    // Find and remove access time entries for KeyExtents
-    // that are no longer in the onlineTablets collection
-    Set<KeyExtent> missing = onDemandTabletAccessTimes.keySet().stream()
-        .filter(k -> !online.containsKey(k)).collect(Collectors.toSet());
-    if (!missing.isEmpty()) {
-      log.debug("Removing onDemandAccessTimes for tablets as tablets no longer online: {}",
-          missing);
-      missing.forEach(onDemandTabletAccessTimes::remove);
-      if (onDemandTabletAccessTimes.isEmpty()) {
-        return;
-      }
-    }
-
-    // It's possible, from a tablet split or merge for example,
-    // that there is an on-demand tablet that is hosted for which
-    // we have no access time. Add any missing online on-demand
-    // tablets
-    online.forEach((k, v) -> {
-      if (v.isOnDemand() && !onDemandTabletAccessTimes.containsKey(k)) {
-        insertOnDemandAccessTime(k);
+    // Sort the extents so that we can process them by table.
+    final SortedMap<KeyExtent,Long> sortedOnDemandExtents = new TreeMap<>();
+    // We only want to operate on OnDemand Tablets
+    online.entrySet().forEach((e) -> {
+      if (e.getValue().isOnDemand()) {
+        sortedOnDemandExtents.put(e.getKey(), e.getValue().getLastAccessTime());
       }
     });
 
-    log.debug("Evaluating online on-demand tablets: {}", onDemandTabletAccessTimes);
-
-    if (onDemandTabletAccessTimes.isEmpty()) {
+    if (sortedOnDemandExtents.isEmpty()) {
       return;
     }
 
+    log.debug("Evaluating online on-demand tablets: {}", sortedOnDemandExtents);
+
     // If the TabletServer is running low on memory, don't call the SPI
     // plugin to evaluate which on-demand tablets to unload, just get the
     // on-demand tablet with the oldest access time and unload it.
     if (getContext().getLowMemoryDetector().isRunningLowOnMemory()) {
       final SortedMap<Long,KeyExtent> timeSortedOnDemandExtents = new TreeMap<>();
-      onDemandTabletAccessTimes.forEach((k, v) -> timeSortedOnDemandExtents.put(v.get(), k));
+      sortedOnDemandExtents.forEach((k, v) -> timeSortedOnDemandExtents.put(v, k));
       Long oldestAccessTime = timeSortedOnDemandExtents.firstKey();

Review Comment:
   Implemented in a7361ea



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Track lastAccessTime in Tablet, not in TabletServer collection [accumulo]

Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on code in PR #3855:
URL: https://github.com/apache/accumulo/pull/3855#discussion_r1364155487


##########
server/tserver/src/main/java/org/apache/accumulo/tserver/AssignmentHandler.java:
##########
@@ -190,6 +189,7 @@ public void run() {
           server.recentlyUnloadedCache.remove(tablet.getExtent());
         }
       }
+      tablet.setLastAccessTime();

Review Comment:
   This may not be needed if this can be set during tablet creation. If it is needed, should probably be done before the tablet is added to online tablets.  Once its added to online tablets it can be considered for unload.



##########
server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java:
##########
@@ -1182,41 +1156,27 @@ public void evaluateOnDemandTabletsForUnload() {
 
     final SortedMap<KeyExtent,Tablet> online = getOnlineTablets();
 
-    // Find and remove access time entries for KeyExtents
-    // that are no longer in the onlineTablets collection
-    Set<KeyExtent> missing = onDemandTabletAccessTimes.keySet().stream()
-        .filter(k -> !online.containsKey(k)).collect(Collectors.toSet());
-    if (!missing.isEmpty()) {
-      log.debug("Removing onDemandAccessTimes for tablets as tablets no longer online: {}",
-          missing);
-      missing.forEach(onDemandTabletAccessTimes::remove);
-      if (onDemandTabletAccessTimes.isEmpty()) {
-        return;
-      }
-    }
-
-    // It's possible, from a tablet split or merge for example,
-    // that there is an on-demand tablet that is hosted for which
-    // we have no access time. Add any missing online on-demand
-    // tablets
-    online.forEach((k, v) -> {
-      if (v.isOnDemand() && !onDemandTabletAccessTimes.containsKey(k)) {
-        insertOnDemandAccessTime(k);
+    // Sort the extents so that we can process them by table.
+    final SortedMap<KeyExtent,Long> sortedOnDemandExtents = new TreeMap<>();
+    // We only want to operate on OnDemand Tablets
+    online.entrySet().forEach((e) -> {
+      if (e.getValue().isOnDemand()) {
+        sortedOnDemandExtents.put(e.getKey(), e.getValue().getLastAccessTime());
       }
     });
 
-    log.debug("Evaluating online on-demand tablets: {}", onDemandTabletAccessTimes);
-
-    if (onDemandTabletAccessTimes.isEmpty()) {
+    if (sortedOnDemandExtents.isEmpty()) {
       return;
     }
 
+    log.debug("Evaluating online on-demand tablets: {}", sortedOnDemandExtents);
+
     // If the TabletServer is running low on memory, don't call the SPI
     // plugin to evaluate which on-demand tablets to unload, just get the
     // on-demand tablet with the oldest access time and unload it.
     if (getContext().getLowMemoryDetector().isRunningLowOnMemory()) {
       final SortedMap<Long,KeyExtent> timeSortedOnDemandExtents = new TreeMap<>();
-      onDemandTabletAccessTimes.forEach((k, v) -> timeSortedOnDemandExtents.put(v.get(), k));
+      sortedOnDemandExtents.forEach((k, v) -> timeSortedOnDemandExtents.put(v, k));
       Long oldestAccessTime = timeSortedOnDemandExtents.firstKey();

Review Comment:
   This is an existing issue, but could fix it here.   Think this is sorting on nano time which is not correct because nano time has an arbitrary starting point in a process and it can wrap.  So should find the largest difference.
   
   ```suggestion
         long currTime = System.nanoTime();
         sortedOnDemandExtents.forEach((k, v) -> timeSortedOnDemandExtents.put(v-currTime, k));
         Long oldestAccessTime = timeSortedOnDemandExtents.lasttKey();
   ```



##########
server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java:
##########
@@ -178,6 +178,8 @@ enum CompactionState {
 
   private final int logId;
 
+  private volatile long lastAccessTime = 0;

Review Comment:
   Can this be set at creation?
   
   ```suggestion
     private volatile long lastAccessTime = System.nanoTime();
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Track lastAccessTime in Tablet, not in TabletServer collection [accumulo]

Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion merged PR #3855:
URL: https://github.com/apache/accumulo/pull/3855


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org