[GitHub] [hadoop] steveloughran commented on a diff in pull request #5133: HADOOP-18521. Draft change - ABFS Prefetch corruption

[GitBox <gi...@apache.org>]

steveloughran commented on code in PR #5133:
URL: https://github.com/apache/hadoop/pull/5133#discussion_r1031522202


##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManager.java:
##########
@@ -247,7 +247,7 @@ private synchronized boolean tryEvict() {
 
     // first, try buffers where all bytes have been consumed (approximated as first and last bytes consumed)
     for (ReadBuffer buf : completedReadList) {
-      if (buf.isFirstByteConsumed() && buf.isLastByteConsumed()) {
+      if (buf.getStream().isClosed() || (buf.isFirstByteConsumed() && buf.isLastByteConsumed())) {

Review Comment:
   this doesn't quite do the right thing as evict() is looking for a completed read with an allocated buffer. here it will also find any completed read whose buffer was released prematurely.
   
   i'm picking up this change in my pr, but including the check for having the buffer, so it will *only* pick up those records whose stream was closed after a successful read completed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org