[jira] [Commented] (HADOOP-18546) disable purging list of in progress reads in abfs stream closed

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17641644#comment-17641644 ] 

ASF GitHub Bot commented on HADOOP-18546:
-----------------------------------------

pranavsaxena-microsoft opened a new pull request, #5176:
URL: https://github.com/apache/hadoop/pull/5176

   JIRA: https://issues.apache.org/jira/browse/HADOOP-18546
   **Details:**
   AbfsInputStream.close() can trigger the return of buffers used for active prefetch GET requests into the ReadBufferManager free buffer pool.
   
   A subsequent prefetch by a different stream in the same process may acquire this same buffer. This can lead to risk of corruption of its own prefetched data, data which may then be returned to that other thread.
   Parent JIRA: https://issues.apache.org/jira/browse/HADOOP-18521
   
   In this PR, we are disabling the purging of the inprogressList. The readBuffers in InProgressList will get to ReadBufferWorker and get processed and finally get into completedList. After a thresholdAgeMilliseconds, the readBuffer would be evicted (https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManager.java#L280-L285)
   
   **Testing:**
   https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManager.java#L280-L285




> disable purging list of in progress reads in abfs stream closed
> ---------------------------------------------------------------
>
>                 Key: HADOOP-18546
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18546
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>    Affects Versions: 3.3.4
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>
> turn off the prune of in progress reads in ReadBufferManager::purgeBuffersForStream
> this will ensure active prefetches for a closed stream complete. they wiill then get to the completed list and hang around until evicted by timeout, but at least prefetching will be safe.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org