[GitHub] [hadoop] snvijaya commented on a diff in pull request #5176: HADOOP-18546. ABFS:disable purging list of in progress reads in abfs stream closed

[GitBox <gi...@apache.org>]

snvijaya commented on code in PR #5176:
URL: https://github.com/apache/hadoop/pull/5176#discussion_r1037025160


##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java:
##########
@@ -495,6 +509,199 @@ public void testSuccessfulReadAhead() throws Exception {
     checkEvictedStatus(inputStream, 0, true);
   }
 
+  /**
+   * This test expects InProgressList is not purged by the inputStream close.
+   * The readBuffer will move to completedList and then finally should get evicted.
+   */
+  @Test
+  public void testStreamPurgeDuringReadAheadCallExecuting() throws Exception {
+    AbfsClient client = getMockAbfsClient();
+    AbfsRestOperation successOp = getMockRestOp();
+
+    final AtomicInteger movedToInProgressList = new AtomicInteger(0);
+    final AtomicInteger movedToCompletedList = new AtomicInteger(0);
+    final AtomicBoolean preClosedAssertion = new AtomicBoolean(false);
+
+    Mockito.doAnswer(invocationOnMock -> {
+          movedToInProgressList.incrementAndGet();
+          while (movedToInProgressList.get() < 3 || !preClosedAssertion.get()) {
+
+          }
+          movedToCompletedList.incrementAndGet();
+          return successOp;
+        })
+        .when(client)

Review Comment:
   The test is trying to unit test a bigger scope of existing inprogress buffer moving to completed list. Will be nice to scope the test to inProgressList and freelist counts, before and after close. 
   
   At this client.read() mock, I would suggest mocks that will invoke a large sleep for each read. That way after queueReadAheads call and a 1 sec sleep, 3 buffers will be stuck inProgessList and the freeeList should show 13 free. The asserts should continue to hold to same numbers post close as well.



##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java:
##########
@@ -495,6 +509,199 @@ public void testSuccessfulReadAhead() throws Exception {
     checkEvictedStatus(inputStream, 0, true);
   }
 
+  /**
+   * This test expects InProgressList is not purged by the inputStream close.
+   * The readBuffer will move to completedList and then finally should get evicted.
+   */
+  @Test
+  public void testStreamPurgeDuringReadAheadCallExecuting() throws Exception {
+    AbfsClient client = getMockAbfsClient();
+    AbfsRestOperation successOp = getMockRestOp();
+
+    final AtomicInteger movedToInProgressList = new AtomicInteger(0);
+    final AtomicInteger movedToCompletedList = new AtomicInteger(0);
+    final AtomicBoolean preClosedAssertion = new AtomicBoolean(false);
+
+    Mockito.doAnswer(invocationOnMock -> {
+          movedToInProgressList.incrementAndGet();
+          while (movedToInProgressList.get() < 3 || !preClosedAssertion.get()) {
+
+          }
+          movedToCompletedList.incrementAndGet();
+          return successOp;
+        })
+        .when(client)
+        .read(any(String.class), any(Long.class), any(byte[].class),
+            any(Integer.class), any(Integer.class), any(String.class),
+            any(String.class), any(TracingContext.class));
+
+    AbfsInputStream inputStream = getAbfsInputStream(client,
+        "testSuccessfulReadAhead.txt");
+    queueReadAheads(inputStream);
+
+    final ReadBufferManager readBufferManager
+        = ReadBufferManager.getBufferManager();
+    while (movedToInProgressList.get() < 3) {
+
+    }
+    Assertions.assertThat(
+            getStreamRelatedBufferCount(readBufferManager.getInProgressCopiedList(),
+                inputStream))
+        .describedAs("InProgressList should have 3 elements")
+        .isEqualTo(3);
+    Assertions.assertThat(getStreamRelatedBufferCount(
+            readBufferManager.getCompletedReadListCopy(), inputStream))
+        .describedAs("CompletedList should have 3 elements")
+        .isEqualTo(0);
+
+    inputStream.close();
+
+    Assertions.assertThat(
+            getStreamRelatedBufferCount(readBufferManager.getInProgressCopiedList(),
+                inputStream))
+        .describedAs("InProgressList should have 3 elements")
+        .isEqualTo(3);
+    Assertions.assertThat(getStreamRelatedBufferCount(
+            readBufferManager.getCompletedReadListCopy(), inputStream))
+        .describedAs("CompletedList should have 3 elements")
+        .isEqualTo(0);
+    preClosedAssertion.set(true);
+
+    while (movedToCompletedList.get() < 3) {
+
+    }
+
+    //Sleep so that response from mockedClient gets back to ReadBufferWorker and
+    // can populate into completedList.
+    Thread.sleep(10000l);
+
+    Assertions.assertThat(getStreamRelatedBufferCount(
+            readBufferManager.getCompletedReadListCopy(), inputStream))
+        .describedAs("CompletedList should have 3 elements")
+        .isEqualTo(3);
+
+    Thread.sleep(readBufferManager.getThresholdAgeMilliseconds());
+
+    readBufferManager.callTryEvict();
+    readBufferManager.callTryEvict();
+    readBufferManager.callTryEvict();
+
+    Assertions.assertThat(getStreamRelatedBufferCount(
+            readBufferManager.getCompletedReadListCopy(), inputStream))
+        .describedAs("CompletedList should have 0 elements")
+        .isEqualTo(0);
+  }
+
+
+  /**
+   * This test expects InProgressList is not purged by the inputStream close.
+   * The already readBuffer present in the completedList shall be purged by the
+   * inputStream close.
+   * The readBuffer from inProgressList will move to completedList and then
+   * finally should get evicted.
+   */
+  @Test
+  public void testStreamPurgeDuringReadAheadCallExecutingWithSomeCompletedBuffers()

Review Comment:
   This test seems to be validating effect of purge on completedList.  Does this validate any test scenario that is not already covered in HADOOP-17156 commit testcases ? 
   
   Also, do all test asserts by HADOOP-17156 still hold good after this PR change preventing inprogress list purge ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org