[GitHub] [hadoop] steveloughran commented on a diff in pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.

["steveloughran (via GitHub)" <gi...@apache.org>]

steveloughran commented on code in PR #5832:
URL: https://github.com/apache/hadoop/pull/5832#discussion_r1272698192


##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/CachingBlockManager.java:
##########
@@ -298,16 +306,26 @@ public void requestPrefetch(int blockNumber) {
 
   /**
    * Requests cancellation of any previously issued prefetch requests.
+   * If the reason was switching to random IO, any active prefetched blocks
+   * are still cached.
+   * @param reason why?
    */
   @Override
-  public void cancelPrefetches() {
+  public void cancelPrefetches(final CancelReason reason) {
+    LOG.debug("Cancelling prefetches {}", reason);
     BlockOperations.Operation op = ops.cancelPrefetches();
 
-    for (BufferData data : bufferPool.getAll()) {
-      // We add blocks being prefetched to the local cache so that the prefetch is not wasted.
-      if (data.stateEqualsOneOf(BufferData.State.PREFETCHING, BufferData.State.READY)) {
-        requestCaching(data);
+    if (reason == CancelReason.RandomIO) {
+      for (BufferData data : bufferPool.getAll()) {
+        // We add blocks being prefetched to the local cache so that the prefetch is not wasted.
+        // this only done if the reason is random IO-related, not due to close/unbuffer
+        if (data.stateEqualsOneOf(BufferData.State.PREFETCHING, BufferData.State.READY)) {
+          requestCaching(data);
+        }
       }
+    } else {
+      // free the buffers
+      bufferPool.getAll().forEach(BufferData::setDone);

Review Comment:
   we had so much grief with the abfs prefetch premature release bug that I am scared now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org