You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "m1a2st (via GitHub)" <gi...@apache.org> on 2025/03/16 04:00:04 UTC

[PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

m1a2st opened a new pull request, #19214:
URL: https://github.com/apache/kafka/pull/19214

   The `lastOffset` includes the entire batch header, so we should check `baseOffset` instead.  
   
   To optimize this, we need to update the search logic. The previous approach simply checked whether each batch's `lastOffset()` was greater than or equal to the target offset. Once it found the first batch that met this condition, it returned that batch immediately.  
   
   Now that we are using `baseOffset()`, we need to handle a special case: if the `targetOffset` falls between the `lastOffset` of the previous batch and the `baseOffset` of the matching batch, we should select the matching batch. The updated logic is structured as follows:  
   
   1. First, check if `baseOffset() == targetOffset` for an exact match.  
   2. Keep track of the previous batch (`prevBatch`) for comparison.  
   3. When encountering a batch where `baseOffset() > targetOffset`:  
      - If there is no previous batch, return the current batch.  
      - If the previous batch's `lastOffset()` is greater than or equal to `targetOffset`, return the previous batch (indicating that the target falls within it).  
      - Otherwise, return the current batch.  
   4. After iterating through all batches, check if the last batch contains the target offset.
   
   Test: Verifying Memory Usage Improvement  
   To evaluate whether this optimization helps, I followed the steps below to monitor memory usage:  
   
   1. Start a Standalone Kafka Server  
   ```sh
   KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
   bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties
   bin/kafka-server-start.sh config/server.properties
   ```  
   
   2. Use Performance Console Tools to Produce and Consume Records  
   **Produce Records:**  
   ```sh
   ./kafka-producer-perf-test.sh \
     --topic test-topic \
     --num-records 1000000000 \
     --record-size 100 \
     --throughput -1 \
     --producer-props bootstrap.servers=localhost:9092
   ```  
   **Consume Records:**  
   ```sh
   ./bin/kafka-consumer-perf-test.sh \
     --topic test-topic \
     --messages 1000000000 \
     --bootstrap-server localhost:9092
   ```  
   trunk:
   ![CleanShot 2025-03-16 at 11 53 31@2x](https://github.com/user-attachments/assets/eec26b1d-38ed-41c8-8c49-e5c68643761b)
   this PR:
   ![CleanShot 2025-03-16 at 11 54 05@2x](https://github.com/user-attachments/assets/3857fe7a-8deb-42ff-b7eb-cc11142b74ce)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r2005945059


##########
clients/src/main/java/org/apache/kafka/common/record/FileRecords.java:
##########
@@ -292,17 +292,43 @@ public int writeTo(TransferableChannel destChannel, int offset, int length) thro
     /**
      * Search forward for the file position of the message batch whose last offset that is greater
      * than or equal to the target offset. If no such batch is found, return null.
+     * <p>
+     * The following logic is intentionally designed to minimize memory usage
+     * by avoiding unnecessary calls to {@link FileChannelRecordBatch#lastOffset()} for every batch.
+     * Instead, we use {@link FileChannelRecordBatch#baseOffset()} comparisons when possible, and only 
+     * check {@link FileChannelRecordBatch#lastOffset()} when absolutely necessary.

Review Comment:
   The javadoc should explain the behavior of the method. This type of comment should be next to the actual code - it doesn't affect callers. It likely won't be seen by people changing the implementation either.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on PR #19214:
URL: https://github.com/apache/kafka/pull/19214#issuecomment-2729022427

   Hello @frankvicky, I don’t think we need to update the Javadoc. We are still finding "the file position of the message batch whose last offset is greater than or equal to the target offset." This behavior remains unchanged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r2000915463


##########
clients/src/test/java/org/apache/kafka/common/record/FileRecordsTest.java:
##########
@@ -518,6 +523,176 @@ public void testBytesLengthOfWriteTo() throws IOException {
         verify(channel).transferFrom(any(), anyLong(), eq((long) size - firstWritten));
     }
 
+    /**
+     * Test two condition
+     * 1. If the target offset equals to the base offset of the first batch
+     * 2. If the target offset < the base offset of the first batch
+     */
+    @ParameterizedTest
+    @ValueSource(longs = {5, 10})
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchMatch(long baseOffset) throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch.baseOffset()).thenReturn(baseOffset);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(5L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch), result);
+        verify(batch, never()).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchLastOffsetMatch() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch.baseOffset()).thenReturn(3L);
+        when(batch.lastOffset()).thenReturn(5L);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(5L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch), result);
+        // target is equals to the last offset of the batch, we should call lastOffset
+        verify(batch, times(1)).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountLastBatchLastOffsetMatch() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch prevBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(prevBatch.baseOffset()).thenReturn(5L);
+        when(prevBatch.lastOffset()).thenReturn(12L);
+        FileLogInputStream.FileChannelRecordBatch currentBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(currentBatch.baseOffset()).thenReturn(15L);
+        when(currentBatch.lastOffset()).thenReturn(20L);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, prevBatch, currentBatch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(20L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(currentBatch), result);
+        // Because the target offset is in the current batch, we should not call lastOffset in the previous batch
+        verify(prevBatch, never()).lastOffset();
+        verify(currentBatch, times(1)).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountPrevBatchMatches() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch prevBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(prevBatch.baseOffset()).thenReturn(5L);
+        when(prevBatch.lastOffset()).thenReturn(12L); // > targetOffset
+        FileLogInputStream.FileChannelRecordBatch currentBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(currentBatch.baseOffset()).thenReturn(15L); // >= targetOffset
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, prevBatch, currentBatch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(10L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(prevBatch), result);
+        // Because the target offset is in the current batch, we should call lastOffset 
+        // on the previous batch
+        verify(prevBatch, times(1)).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountAllBatchesSmallerLastBatchDoesntMatch() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch1 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch1.baseOffset()).thenReturn(5L);  // < targetOffset
+        FileLogInputStream.FileChannelRecordBatch batch2 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch2.baseOffset()).thenReturn(8L);  // < targetOffset
+        when(batch2.lastOffset()).thenReturn(9L);  // < targetOffset
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch1, batch2);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(10L, 0);
+
+        assertNull(result);
+        // Because the target offset is exceeded by the last offset of the batch2, 
+        // we should call lastOffset on the batch2
+        verify(batch1, never()).lastOffset();
+        verify(batch2, times(1)).lastOffset();
+    }
+
+    /**
+     * Test two condition
+     * 1. If the target offset < the base offset of the last batch
+     * 2. If the target offset equals to the base offset of the last batch
+     */
+    @ParameterizedTest
+    @ValueSource(longs = {8, 10})
+    public void testSearchForOffsetWithSizeLastOffsetCallCountAllBatchesSmallerLastBatchMatches(long baseOffset) throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch1 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch1.baseOffset()).thenReturn(5L);  // < targetOffset
+        FileLogInputStream.FileChannelRecordBatch batch2 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch2.baseOffset()).thenReturn(baseOffset);  // < targetOffset or == targetOffset
+        when(batch2.lastOffset()).thenReturn(12L); // >= targetOffset
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch1, batch2);
+
+        long targetOffset = 10L;
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(targetOffset, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch2), result);
+        if (targetOffset == baseOffset) {
+            // Because the target offset is equal to the base offset of the batch2, we should not call
+            // lastOffset on batch2
+            verify(batch1, times(1)).lastOffset();
+            verify(batch2, never()).lastOffset();
+        } else {
+            // Because the target offset is in the batch2, we should not call 
+            // lastOffset on batch1
+            verify(batch1, never()).lastOffset();
+            verify(batch2, times(1)).lastOffset();
+        }
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountTargetBetweenTwoBatches() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch1 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch1.baseOffset()).thenReturn(5L);  
+        when(batch1.lastOffset()).thenReturn(10L); 
+        FileLogInputStream.FileChannelRecordBatch batch2 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch2.baseOffset()).thenReturn(15L);  
+        when(batch2.lastOffset()).thenReturn(20L);  
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch1, batch2);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(13L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch2), result);
+        // Because the target offset is between the two batches, we should call lastOffset on the batch1
+        verify(batch1, times(1)).lastOffset();
+        verify(batch2, never()).lastOffset();
+    }
+
+    private void mockFileRecordBatches(FileRecords fileRecords, FileLogInputStream.FileChannelRecordBatch... batch) {
+        List<FileLogInputStream.FileChannelRecordBatch> batches = new ArrayList<>(asList(batch));

Review Comment:
   You're right! Thanks for pointing that out.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #19214:
URL: https://github.com/apache/kafka/pull/19214#issuecomment-2727211778

   @m1a2st Could you please test the approach which I left on the jira? Also, please add unit test to ensure the new approach does NOT call last offset for each batches!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r2003894741


##########
clients/src/main/java/org/apache/kafka/common/record/FileRecords.java:
##########
@@ -297,12 +297,33 @@ public int writeTo(TransferableChannel destChannel, int offset, int length) thro
      * @param startingPosition The starting position in the file to begin searching from.
      * @return the batch's base offset, its physical position, and its size (including log overhead)
      */
-    public LogOffsetPosition searchForOffsetWithSize(long targetOffset, int startingPosition) {
+    public LogOffsetPosition searchForOffsetFromPosition(long targetOffset, int startingPosition) {

Review Comment:
   Please add comments to highlight the following complex logic is to save memory by avoiding using `lastOffset`. Otherwise, there may be a developer who want to streamline the code by using `lastOffset`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r2005945059


##########
clients/src/main/java/org/apache/kafka/common/record/FileRecords.java:
##########
@@ -292,17 +292,43 @@ public int writeTo(TransferableChannel destChannel, int offset, int length) thro
     /**
      * Search forward for the file position of the message batch whose last offset that is greater
      * than or equal to the target offset. If no such batch is found, return null.
+     * <p>
+     * The following logic is intentionally designed to minimize memory usage
+     * by avoiding unnecessary calls to {@link FileChannelRecordBatch#lastOffset()} for every batch.
+     * Instead, we use {@link FileChannelRecordBatch#baseOffset()} comparisons when possible, and only 
+     * check {@link FileChannelRecordBatch#lastOffset()} when absolutely necessary.

Review Comment:
   The javadoc should explain the behavior of the method. This type of comment should be next to the actual code - it doesn't affect callers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r1999448126


##########
clients/src/test/java/org/apache/kafka/common/record/FileRecordsTest.java:
##########
@@ -518,6 +523,176 @@ public void testBytesLengthOfWriteTo() throws IOException {
         verify(channel).transferFrom(any(), anyLong(), eq((long) size - firstWritten));
     }
 
+    /**
+     * Test two condition
+     * 1. If the target offset equals to the base offset of the first batch
+     * 2. If the target offset < the base offset of the first batch
+     */
+    @ParameterizedTest
+    @ValueSource(longs = {5, 10})
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchMatch(long baseOffset) throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch.baseOffset()).thenReturn(baseOffset);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(5L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch), result);
+        verify(batch, never()).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchLastOffsetMatch() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch.baseOffset()).thenReturn(3L);
+        when(batch.lastOffset()).thenReturn(5L);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(5L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch), result);
+        // target is equals to the last offset of the batch, we should call lastOffset
+        verify(batch, times(1)).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountLastBatchLastOffsetMatch() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch prevBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(prevBatch.baseOffset()).thenReturn(5L);
+        when(prevBatch.lastOffset()).thenReturn(12L);
+        FileLogInputStream.FileChannelRecordBatch currentBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(currentBatch.baseOffset()).thenReturn(15L);
+        when(currentBatch.lastOffset()).thenReturn(20L);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, prevBatch, currentBatch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(20L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(currentBatch), result);
+        // Because the target offset is in the current batch, we should not call lastOffset in the previous batch
+        verify(prevBatch, never()).lastOffset();
+        verify(currentBatch, times(1)).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountPrevBatchMatches() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch prevBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(prevBatch.baseOffset()).thenReturn(5L);
+        when(prevBatch.lastOffset()).thenReturn(12L); // > targetOffset
+        FileLogInputStream.FileChannelRecordBatch currentBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(currentBatch.baseOffset()).thenReturn(15L); // >= targetOffset
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, prevBatch, currentBatch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(10L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(prevBatch), result);
+        // Because the target offset is in the current batch, we should call lastOffset 
+        // on the previous batch
+        verify(prevBatch, times(1)).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountAllBatchesSmallerLastBatchDoesntMatch() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch1 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch1.baseOffset()).thenReturn(5L);  // < targetOffset
+        FileLogInputStream.FileChannelRecordBatch batch2 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch2.baseOffset()).thenReturn(8L);  // < targetOffset
+        when(batch2.lastOffset()).thenReturn(9L);  // < targetOffset
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch1, batch2);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(10L, 0);
+
+        assertNull(result);
+        // Because the target offset is exceeded by the last offset of the batch2, 
+        // we should call lastOffset on the batch2
+        verify(batch1, never()).lastOffset();
+        verify(batch2, times(1)).lastOffset();
+    }
+
+    /**
+     * Test two condition
+     * 1. If the target offset < the base offset of the last batch
+     * 2. If the target offset equals to the base offset of the last batch
+     */
+    @ParameterizedTest
+    @ValueSource(longs = {8, 10})
+    public void testSearchForOffsetWithSizeLastOffsetCallCountAllBatchesSmallerLastBatchMatches(long baseOffset) throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch1 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch1.baseOffset()).thenReturn(5L);  // < targetOffset
+        FileLogInputStream.FileChannelRecordBatch batch2 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch2.baseOffset()).thenReturn(baseOffset);  // < targetOffset or == targetOffset
+        when(batch2.lastOffset()).thenReturn(12L); // >= targetOffset
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch1, batch2);
+
+        long targetOffset = 10L;
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(targetOffset, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch2), result);
+        if (targetOffset == baseOffset) {
+            // Because the target offset is equal to the base offset of the batch2, we should not call
+            // lastOffset on batch2
+            verify(batch1, times(1)).lastOffset();

Review Comment:
   > We could optimize this case to avoid calling lastOffset(), right?
   
   Indeed, in the preceding version, this early return was present.. see https://github.com/apache/kafka/pull/19214#discussion_r1999093157
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r2002088337


##########
clients/src/main/java/org/apache/kafka/common/record/FileRecords.java:
##########
@@ -297,12 +297,33 @@ public int writeTo(TransferableChannel destChannel, int offset, int length) thro
      * @param startingPosition The starting position in the file to begin searching from.
      * @return the batch's base offset, its physical position, and its size (including log overhead)
      */
-    public LogOffsetPosition searchForOffsetWithSize(long targetOffset, int startingPosition) {
+    public LogOffsetPosition searchForOffsetFromPosition(long targetOffset, int startingPosition) {
+        FileChannelRecordBatch prevBatch = null;
+
         for (FileChannelRecordBatch batch : batchesFrom(startingPosition)) {
-            long offset = batch.lastOffset();
-            if (offset >= targetOffset)
-                return new LogOffsetPosition(batch.baseOffset(), batch.position(), batch.sizeInBytes());
+            // If baseOffset exactly equals targetOffset, return immediately
+            if (batch.baseOffset() == targetOffset) {
+                return LogOffsetPosition.fromBatch(batch);
+            }
+            
+            // If we find the first batch with baseOffset greater than targetOffset
+            if (batch.baseOffset() > targetOffset) {
+                // If the previous batch contains the target
+                if (prevBatch != null && prevBatch.lastOffset() >= targetOffset)
+                    return LogOffsetPosition.fromBatch(prevBatch);
+                else {
+                    // If there's no previous batch or the previous batch doesn't contain the 
+                    // target, return the current batch
+                    return LogOffsetPosition.fromBatch(batch);
+                }
+            }
+            prevBatch = batch;
         }
+        // Only one case would reach here: all batches have baseOffset less than or equal to targetOffset

Review Comment:
   less than or equal to => less than



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r2005977385


##########
clients/src/main/java/org/apache/kafka/common/record/FileRecords.java:
##########
@@ -292,17 +292,43 @@ public int writeTo(TransferableChannel destChannel, int offset, int length) thro
     /**
      * Search forward for the file position of the message batch whose last offset that is greater
      * than or equal to the target offset. If no such batch is found, return null.
+     * <p>
+     * The following logic is intentionally designed to minimize memory usage
+     * by avoiding unnecessary calls to {@link FileChannelRecordBatch#lastOffset()} for every batch.
+     * Instead, we use {@link FileChannelRecordBatch#baseOffset()} comparisons when possible, and only 
+     * check {@link FileChannelRecordBatch#lastOffset()} when absolutely necessary.

Review Comment:
   @m1a2st could you please file a minor to address it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r1999450816


##########
clients/src/main/java/org/apache/kafka/common/record/FileRecords.java:
##########
@@ -298,11 +298,28 @@ public int writeTo(TransferableChannel destChannel, int offset, int length) thro
      * @return the batch's base offset, its physical position, and its size (including log overhead)
      */
     public LogOffsetPosition searchForOffsetWithSize(long targetOffset, int startingPosition) {
+        FileChannelRecordBatch prevBatch = null;
+
         for (FileChannelRecordBatch batch : batchesFrom(startingPosition)) {
-            long offset = batch.lastOffset();
-            if (offset >= targetOffset)
-                return new LogOffsetPosition(batch.baseOffset(), batch.position(), batch.sizeInBytes());
+            // This indicates that either the current batch or the previous batch 
+            // contains the target we are looking for.
+            if (batch.baseOffset() >= targetOffset) {

Review Comment:
   @m1a2st it appears the previous version provided greater clarity and optimization. Hence, please add the case `if (batch.baseOffset() == targetOffset) ` back.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r1998877164


##########
clients/src/main/java/org/apache/kafka/common/record/FileRecords.java:
##########
@@ -464,6 +481,12 @@ public static class LogOffsetPosition {
         public final int position;
         public final int size;
 
+        public LogOffsetPosition(FileChannelRecordBatch batch) {

Review Comment:
   Since this LogOffsetPosition is used like a value class, this might be better as an factory method like `LogOffsetPosition#fromBatch` instead of an alternate constructor



##########
clients/src/main/java/org/apache/kafka/common/record/FileRecords.java:
##########
@@ -298,11 +298,28 @@ public int writeTo(TransferableChannel destChannel, int offset, int length) thro
      * @return the batch's base offset, its physical position, and its size (including log overhead)
      */
     public LogOffsetPosition searchForOffsetWithSize(long targetOffset, int startingPosition) {
+        FileChannelRecordBatch prevBatch = null;
+
         for (FileChannelRecordBatch batch : batchesFrom(startingPosition)) {
-            long offset = batch.lastOffset();
-            if (offset >= targetOffset)
-                return new LogOffsetPosition(batch.baseOffset(), batch.position(), batch.sizeInBytes());
+            // This indicates that either the current batch or the previous batch 
+            // contains the target we are looking for.
+            if (batch.baseOffset() >= targetOffset) {
+                // Check if the previous batch contains the target
+                if (prevBatch != null && prevBatch.lastOffset() >= targetOffset)
+                    return new LogOffsetPosition(prevBatch);
+                else
+                    // If there's no previous batch or the previous batch doesn't contain the 
+                    // target, return the current batch
+                    return new LogOffsetPosition(batch);

Review Comment:
   nit: please add curly braces here



##########
clients/src/main/java/org/apache/kafka/common/record/FileRecords.java:
##########
@@ -298,11 +298,28 @@ public int writeTo(TransferableChannel destChannel, int offset, int length) thro
      * @return the batch's base offset, its physical position, and its size (including log overhead)
      */
     public LogOffsetPosition searchForOffsetWithSize(long targetOffset, int startingPosition) {
+        FileChannelRecordBatch prevBatch = null;
+
         for (FileChannelRecordBatch batch : batchesFrom(startingPosition)) {
-            long offset = batch.lastOffset();
-            if (offset >= targetOffset)
-                return new LogOffsetPosition(batch.baseOffset(), batch.position(), batch.sizeInBytes());
+            // This indicates that either the current batch or the previous batch 
+            // contains the target we are looking for.
+            if (batch.baseOffset() >= targetOffset) {

Review Comment:
   Why are we using `>=` here? This means we need to deal with "current batch" and "prev batch" in the same loop iteration. This makes the code a bit more difficult to understand. Would it be simpler to only use `>` and look at the "prev batch" ? 
   
   Maybe I'm missing something that makes this necessary.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r1997640867


##########
clients/src/main/java/org/apache/kafka/common/record/FileRecords.java:
##########
@@ -298,11 +298,26 @@ public int writeTo(TransferableChannel destChannel, int offset, int length) thro
      * @return the batch's base offset, its physical position, and its size (including log overhead)
      */
     public LogOffsetPosition searchForOffsetWithSize(long targetOffset, int startingPosition) {
+        FileChannelRecordBatch prevBatch = null;
+
         for (FileChannelRecordBatch batch : batchesFrom(startingPosition)) {
-            long offset = batch.lastOffset();
-            if (offset >= targetOffset)
-                return new LogOffsetPosition(batch.baseOffset(), batch.position(), batch.sizeInBytes());
+            if (batch.baseOffset() >= targetOffset) {

Review Comment:
   Could you please add comments to explain why we want to avoid exercising `lastOffset` ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r1999903645


##########
clients/src/main/java/org/apache/kafka/common/record/FileRecords.java:
##########
@@ -298,11 +298,28 @@ public int writeTo(TransferableChannel destChannel, int offset, int length) thro
      * @return the batch's base offset, its physical position, and its size (including log overhead)
      */
     public LogOffsetPosition searchForOffsetWithSize(long targetOffset, int startingPosition) {
+        FileChannelRecordBatch prevBatch = null;
+
         for (FileChannelRecordBatch batch : batchesFrom(startingPosition)) {
-            long offset = batch.lastOffset();
-            if (offset >= targetOffset)
-                return new LogOffsetPosition(batch.baseOffset(), batch.position(), batch.sizeInBytes());
+            // This indicates that either the current batch or the previous batch 
+            // contains the target we are looking for.
+            if (batch.baseOffset() >= targetOffset) {

Review Comment:
   What about a mixture of both versions?
   
   ```java
   for (FileChannelRecordBatch batch : batchesFrom(startingPosition)) {
       // if baseOffset exactly equals targetOffset, return immediately
       if (batch.baseOffset() == targetOffset) {
           return LogOffsetPosition.fromBatch(batch);
       }
   
       // If we find the first batch with baseOffset greater than targetOffset
       if (batch.baseOffset() > targetOffset) {
           // Check if the previous batch contains the target
           if (prevBatch != null && prevBatch.lastOffset() >= targetOffset)
               return LogOffsetPosition.fromBatch(prevBatch);
           else {
               // If there's no previous batch or the previous batch doesn't contain the 
               // target, return the current batch
               return LogOffsetPosition.fromBatch(batch);
           }
       }
       prevBatch = batch;
   }
   ```



##########
clients/src/test/java/org/apache/kafka/common/record/FileRecordsTest.java:
##########
@@ -518,6 +523,176 @@ public void testBytesLengthOfWriteTo() throws IOException {
         verify(channel).transferFrom(any(), anyLong(), eq((long) size - firstWritten));
     }
 
+    /**
+     * Test two condition
+     * 1. If the target offset equals to the base offset of the first batch
+     * 2. If the target offset < the base offset of the first batch
+     */
+    @ParameterizedTest
+    @ValueSource(longs = {5, 10})
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchMatch(long baseOffset) throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch.baseOffset()).thenReturn(baseOffset);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(5L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch), result);
+        verify(batch, never()).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchLastOffsetMatch() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch.baseOffset()).thenReturn(3L);
+        when(batch.lastOffset()).thenReturn(5L);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(5L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch), result);
+        // target is equals to the last offset of the batch, we should call lastOffset
+        verify(batch, times(1)).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountLastBatchLastOffsetMatch() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch prevBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(prevBatch.baseOffset()).thenReturn(5L);
+        when(prevBatch.lastOffset()).thenReturn(12L);
+        FileLogInputStream.FileChannelRecordBatch currentBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(currentBatch.baseOffset()).thenReturn(15L);
+        when(currentBatch.lastOffset()).thenReturn(20L);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, prevBatch, currentBatch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(20L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(currentBatch), result);
+        // Because the target offset is in the current batch, we should not call lastOffset in the previous batch
+        verify(prevBatch, never()).lastOffset();
+        verify(currentBatch, times(1)).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountPrevBatchMatches() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch prevBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(prevBatch.baseOffset()).thenReturn(5L);
+        when(prevBatch.lastOffset()).thenReturn(12L); // > targetOffset
+        FileLogInputStream.FileChannelRecordBatch currentBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(currentBatch.baseOffset()).thenReturn(15L); // >= targetOffset
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, prevBatch, currentBatch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(10L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(prevBatch), result);
+        // Because the target offset is in the current batch, we should call lastOffset 
+        // on the previous batch
+        verify(prevBatch, times(1)).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountAllBatchesSmallerLastBatchDoesntMatch() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch1 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch1.baseOffset()).thenReturn(5L);  // < targetOffset
+        FileLogInputStream.FileChannelRecordBatch batch2 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch2.baseOffset()).thenReturn(8L);  // < targetOffset
+        when(batch2.lastOffset()).thenReturn(9L);  // < targetOffset
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch1, batch2);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(10L, 0);
+
+        assertNull(result);
+        // Because the target offset is exceeded by the last offset of the batch2, 
+        // we should call lastOffset on the batch2
+        verify(batch1, never()).lastOffset();
+        verify(batch2, times(1)).lastOffset();
+    }
+
+    /**
+     * Test two condition
+     * 1. If the target offset < the base offset of the last batch
+     * 2. If the target offset equals to the base offset of the last batch
+     */
+    @ParameterizedTest
+    @ValueSource(longs = {8, 10})
+    public void testSearchForOffsetWithSizeLastOffsetCallCountAllBatchesSmallerLastBatchMatches(long baseOffset) throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch1 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch1.baseOffset()).thenReturn(5L);  // < targetOffset
+        FileLogInputStream.FileChannelRecordBatch batch2 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch2.baseOffset()).thenReturn(baseOffset);  // < targetOffset or == targetOffset
+        when(batch2.lastOffset()).thenReturn(12L); // >= targetOffset
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch1, batch2);
+
+        long targetOffset = 10L;
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(targetOffset, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch2), result);
+        if (targetOffset == baseOffset) {
+            // Because the target offset is equal to the base offset of the batch2, we should not call
+            // lastOffset on batch2
+            verify(batch1, times(1)).lastOffset();
+            verify(batch2, never()).lastOffset();
+        } else {
+            // Because the target offset is in the batch2, we should not call 
+            // lastOffset on batch1
+            verify(batch1, never()).lastOffset();
+            verify(batch2, times(1)).lastOffset();
+        }
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountTargetBetweenTwoBatches() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch1 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch1.baseOffset()).thenReturn(5L);  
+        when(batch1.lastOffset()).thenReturn(10L); 
+        FileLogInputStream.FileChannelRecordBatch batch2 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch2.baseOffset()).thenReturn(15L);  
+        when(batch2.lastOffset()).thenReturn(20L);  
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch1, batch2);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(13L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch2), result);
+        // Because the target offset is between the two batches, we should call lastOffset on the batch1
+        verify(batch1, times(1)).lastOffset();
+        verify(batch2, never()).lastOffset();
+    }
+
+    private void mockFileRecordBatches(FileRecords fileRecords, FileLogInputStream.FileChannelRecordBatch... batch) {
+        List<FileLogInputStream.FileChannelRecordBatch> batches = new ArrayList<>(asList(batch));

Review Comment:
   Out of curiosity, is there a reason not to use the list returned by `asList()`?



##########
clients/src/test/java/org/apache/kafka/common/record/FileRecordsTest.java:
##########
@@ -518,6 +523,176 @@ public void testBytesLengthOfWriteTo() throws IOException {
         verify(channel).transferFrom(any(), anyLong(), eq((long) size - firstWritten));
     }
 
+    /**
+     * Test two condition
+     * 1. If the target offset equals to the base offset of the first batch
+     * 2. If the target offset < the base offset of the first batch
+     */
+    @ParameterizedTest
+    @ValueSource(longs = {5, 10})
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchMatch(long baseOffset) throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch.baseOffset()).thenReturn(baseOffset);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(5L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch), result);
+        verify(batch, never()).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchLastOffsetMatch() throws IOException {

Review Comment:
   I empathize with the challenge of coming up with meaningful, yet succinct test method names. I can't get more than maybe two thirds of the way through that method name before my brain shuts down. But I think that says more about my brain than the method name 😆



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on PR #19214:
URL: https://github.com/apache/kafka/pull/19214#issuecomment-2730090896

   > Also, how about the case where a target offset lies between batches (which I think is possible due to compaction
   
   In the previous logic, it would point to the next batch. For example, if batch1 has a `baseOffset` of 1 and a `lastOffset` of 5, and batch2 has a `baseOffset` of 10 and a `lastOffset` of 15, with a targetOffset of 7, the check `lastOffset >= targetOffset` would cause it to return batch2.  
   
   I have also added a test for this scenario.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r2000920105


##########
clients/src/test/java/org/apache/kafka/common/record/FileRecordsTest.java:
##########
@@ -518,6 +523,176 @@ public void testBytesLengthOfWriteTo() throws IOException {
         verify(channel).transferFrom(any(), anyLong(), eq((long) size - firstWritten));
     }
 
+    /**
+     * Test two condition
+     * 1. If the target offset equals to the base offset of the first batch
+     * 2. If the target offset < the base offset of the first batch
+     */
+    @ParameterizedTest
+    @ValueSource(longs = {5, 10})
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchMatch(long baseOffset) throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch.baseOffset()).thenReturn(baseOffset);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(5L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch), result);
+        verify(batch, never()).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchLastOffsetMatch() throws IOException {

Review Comment:
   I have updated the test names to make them clearer and more understandable. If it's still not clear enough, please feel free to share your suggestions! Thanks :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r2005987706


##########
clients/src/main/java/org/apache/kafka/common/record/FileRecords.java:
##########
@@ -292,17 +292,43 @@ public int writeTo(TransferableChannel destChannel, int offset, int length) thro
     /**
      * Search forward for the file position of the message batch whose last offset that is greater
      * than or equal to the target offset. If no such batch is found, return null.
+     * <p>
+     * The following logic is intentionally designed to minimize memory usage
+     * by avoiding unnecessary calls to {@link FileChannelRecordBatch#lastOffset()} for every batch.
+     * Instead, we use {@link FileChannelRecordBatch#baseOffset()} comparisons when possible, and only 
+     * check {@link FileChannelRecordBatch#lastOffset()} when absolutely necessary.

Review Comment:
   https://github.com/apache/kafka/pull/19257



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 merged PR #19214:
URL: https://github.com/apache/kafka/pull/19214


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r1997641038


##########
clients/src/test/java/org/apache/kafka/common/record/FileRecordsTest.java:
##########
@@ -518,6 +521,52 @@ public void testBytesLengthOfWriteTo() throws IOException {
         verify(channel).transferFrom(any(), anyLong(), eq((long) size - firstWritten));
     }
 
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchMatch() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch.baseOffset()).thenReturn(10L);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+
+        List<FileLogInputStream.FileChannelRecordBatch> batches = new ArrayList<>();
+        batches.add(batch);
+        doReturn((Iterable<FileLogInputStream.FileChannelRecordBatch>) batches::iterator)
+                .when(fileRecords)
+                .batchesFrom(anyInt());
+        
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(5L, 0);
+
+        assertEquals(new FileRecords.LogOffsetPosition(batch), result);
+        verify(batch, never()).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountPrevBatchMatches() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch prevBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(prevBatch.baseOffset()).thenReturn(5L);
+        when(prevBatch.lastOffset()).thenReturn(12L); // > targetOffset
+        FileLogInputStream.FileChannelRecordBatch currentBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(currentBatch.baseOffset()).thenReturn(15L); // >= targetOffset
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        
+        List<FileLogInputStream.FileChannelRecordBatch> batches = new ArrayList<>();
+        batches.add(prevBatch);
+        batches.add(currentBatch);
+        doReturn((Iterable<FileLogInputStream.FileChannelRecordBatch>) batches::iterator)
+                .when(fileRecords)
+                .batchesFrom(anyInt());
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(10L, 0);
+
+        assertEquals(new FileRecords.LogOffsetPosition(prevBatch), result);
+        verify(prevBatch, times(1)).lastOffset();

Review Comment:
   please add comments for this verification.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r1999093157


##########
clients/src/main/java/org/apache/kafka/common/record/FileRecords.java:
##########
@@ -298,11 +298,28 @@ public int writeTo(TransferableChannel destChannel, int offset, int length) thro
      * @return the batch's base offset, its physical position, and its size (including log overhead)
      */
     public LogOffsetPosition searchForOffsetWithSize(long targetOffset, int startingPosition) {
+        FileChannelRecordBatch prevBatch = null;
+
         for (FileChannelRecordBatch batch : batchesFrom(startingPosition)) {
-            long offset = batch.lastOffset();
-            if (offset >= targetOffset)
-                return new LogOffsetPosition(batch.baseOffset(), batch.position(), batch.sizeInBytes());
+            // This indicates that either the current batch or the previous batch 
+            // contains the target we are looking for.
+            if (batch.baseOffset() >= targetOffset) {

Review Comment:
   In my first version, I split the logic into separate parts. However, later I merged the logic to reduce the number of early return branches.
   ```
           for (FileChannelRecordBatch batch : batchesFrom(startingPosition)) {
               // if baseOffset exactly equals targetOffset, return immediately
               if (batch.baseOffset() == targetOffset) {
                   return new LogOffsetPosition(batch);
               }
   
               // If we find the first batch with baseOffset greater than targetOffset
               if (batch.baseOffset() > targetOffset) {
                   // If there's no previous batch, return the current batch
                   if (prevBatch == null) {
                       return new LogOffsetPosition(batch);
                   }
   
                   // Check if the previous batch contains the target
                   if (prevBatch.lastOffset() >= targetOffset) {
                       return new LogOffsetPosition(prevBatch);
                   } else {
                       // If the previous batch doesn't contain the target, return the current batch
                       return new LogOffsetPosition(batch);
                   }
               }
               prevBatch = batch;
           }
    ```
    If having the condition if (batch.baseOffset() == targetOffset) makes it clearer. I can add it back.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r1999287436


##########
clients/src/test/java/org/apache/kafka/common/record/FileRecordsTest.java:
##########
@@ -518,6 +523,176 @@ public void testBytesLengthOfWriteTo() throws IOException {
         verify(channel).transferFrom(any(), anyLong(), eq((long) size - firstWritten));
     }
 
+    /**
+     * Test two condition
+     * 1. If the target offset equals to the base offset of the first batch
+     * 2. If the target offset < the base offset of the first batch
+     */
+    @ParameterizedTest
+    @ValueSource(longs = {5, 10})
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchMatch(long baseOffset) throws IOException {

Review Comment:
   Quite a long name. The `LastOffsetCallCount` doesn't seem to convey much.



##########
clients/src/test/java/org/apache/kafka/common/record/FileRecordsTest.java:
##########
@@ -518,6 +523,176 @@ public void testBytesLengthOfWriteTo() throws IOException {
         verify(channel).transferFrom(any(), anyLong(), eq((long) size - firstWritten));
     }
 
+    /**
+     * Test two condition

Review Comment:
   two condition => two conditions



##########
clients/src/test/java/org/apache/kafka/common/record/FileRecordsTest.java:
##########
@@ -518,6 +523,176 @@ public void testBytesLengthOfWriteTo() throws IOException {
         verify(channel).transferFrom(any(), anyLong(), eq((long) size - firstWritten));
     }
 
+    /**
+     * Test two condition
+     * 1. If the target offset equals to the base offset of the first batch
+     * 2. If the target offset < the base offset of the first batch
+     */
+    @ParameterizedTest
+    @ValueSource(longs = {5, 10})
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchMatch(long baseOffset) throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch.baseOffset()).thenReturn(baseOffset);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(5L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch), result);
+        verify(batch, never()).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchLastOffsetMatch() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch.baseOffset()).thenReturn(3L);
+        when(batch.lastOffset()).thenReturn(5L);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(5L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch), result);
+        // target is equals to the last offset of the batch, we should call lastOffset

Review Comment:
   is equals to  => is equal to



##########
clients/src/test/java/org/apache/kafka/common/record/FileRecordsTest.java:
##########
@@ -518,6 +523,176 @@ public void testBytesLengthOfWriteTo() throws IOException {
         verify(channel).transferFrom(any(), anyLong(), eq((long) size - firstWritten));
     }
 
+    /**
+     * Test two condition
+     * 1. If the target offset equals to the base offset of the first batch

Review Comment:
   equals to => is equal to



##########
clients/src/test/java/org/apache/kafka/common/record/FileRecordsTest.java:
##########
@@ -518,6 +523,176 @@ public void testBytesLengthOfWriteTo() throws IOException {
         verify(channel).transferFrom(any(), anyLong(), eq((long) size - firstWritten));
     }
 
+    /**
+     * Test two condition
+     * 1. If the target offset equals to the base offset of the first batch
+     * 2. If the target offset < the base offset of the first batch
+     */
+    @ParameterizedTest
+    @ValueSource(longs = {5, 10})
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchMatch(long baseOffset) throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch.baseOffset()).thenReturn(baseOffset);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(5L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch), result);
+        verify(batch, never()).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchLastOffsetMatch() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch.baseOffset()).thenReturn(3L);
+        when(batch.lastOffset()).thenReturn(5L);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(5L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch), result);
+        // target is equals to the last offset of the batch, we should call lastOffset
+        verify(batch, times(1)).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountLastBatchLastOffsetMatch() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch prevBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(prevBatch.baseOffset()).thenReturn(5L);
+        when(prevBatch.lastOffset()).thenReturn(12L);
+        FileLogInputStream.FileChannelRecordBatch currentBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(currentBatch.baseOffset()).thenReturn(15L);
+        when(currentBatch.lastOffset()).thenReturn(20L);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, prevBatch, currentBatch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(20L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(currentBatch), result);
+        // Because the target offset is in the current batch, we should not call lastOffset in the previous batch
+        verify(prevBatch, never()).lastOffset();
+        verify(currentBatch, times(1)).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountPrevBatchMatches() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch prevBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(prevBatch.baseOffset()).thenReturn(5L);
+        when(prevBatch.lastOffset()).thenReturn(12L); // > targetOffset
+        FileLogInputStream.FileChannelRecordBatch currentBatch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(currentBatch.baseOffset()).thenReturn(15L); // >= targetOffset
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, prevBatch, currentBatch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(10L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(prevBatch), result);
+        // Because the target offset is in the current batch, we should call lastOffset 
+        // on the previous batch
+        verify(prevBatch, times(1)).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountAllBatchesSmallerLastBatchDoesntMatch() throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch1 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch1.baseOffset()).thenReturn(5L);  // < targetOffset
+        FileLogInputStream.FileChannelRecordBatch batch2 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch2.baseOffset()).thenReturn(8L);  // < targetOffset
+        when(batch2.lastOffset()).thenReturn(9L);  // < targetOffset
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch1, batch2);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(10L, 0);
+
+        assertNull(result);
+        // Because the target offset is exceeded by the last offset of the batch2, 
+        // we should call lastOffset on the batch2
+        verify(batch1, never()).lastOffset();
+        verify(batch2, times(1)).lastOffset();
+    }
+
+    /**
+     * Test two condition
+     * 1. If the target offset < the base offset of the last batch
+     * 2. If the target offset equals to the base offset of the last batch
+     */
+    @ParameterizedTest
+    @ValueSource(longs = {8, 10})
+    public void testSearchForOffsetWithSizeLastOffsetCallCountAllBatchesSmallerLastBatchMatches(long baseOffset) throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch1 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch1.baseOffset()).thenReturn(5L);  // < targetOffset
+        FileLogInputStream.FileChannelRecordBatch batch2 = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch2.baseOffset()).thenReturn(baseOffset);  // < targetOffset or == targetOffset
+        when(batch2.lastOffset()).thenReturn(12L); // >= targetOffset
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch1, batch2);
+
+        long targetOffset = 10L;
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(targetOffset, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch2), result);
+        if (targetOffset == baseOffset) {
+            // Because the target offset is equal to the base offset of the batch2, we should not call
+            // lastOffset on batch2
+            verify(batch1, times(1)).lastOffset();

Review Comment:
   We could optimize this case to avoid calling `lastOffset()`, right?



##########
clients/src/main/java/org/apache/kafka/common/record/FileRecords.java:
##########
@@ -298,11 +298,28 @@ public int writeTo(TransferableChannel destChannel, int offset, int length) thro
      * @return the batch's base offset, its physical position, and its size (including log overhead)
      */
     public LogOffsetPosition searchForOffsetWithSize(long targetOffset, int startingPosition) {
+        FileChannelRecordBatch prevBatch = null;

Review Comment:
   This is an existing issue. The name `searchForOffsetWithSize` is a bit confusing. How about sth like `searchForOffsetFromPosition`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r2001161714


##########
clients/src/test/java/org/apache/kafka/common/record/FileRecordsTest.java:
##########
@@ -518,6 +523,176 @@ public void testBytesLengthOfWriteTo() throws IOException {
         verify(channel).transferFrom(any(), anyLong(), eq((long) size - firstWritten));
     }
 
+    /**
+     * Test two condition
+     * 1. If the target offset equals to the base offset of the first batch
+     * 2. If the target offset < the base offset of the first batch
+     */
+    @ParameterizedTest
+    @ValueSource(longs = {5, 10})
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchMatch(long baseOffset) throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch.baseOffset()).thenReturn(baseOffset);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(5L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch), result);
+        verify(batch, never()).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchLastOffsetMatch() throws IOException {

Review Comment:
   I would probably name them sequentially like `testSearchForOffsetWithSize1`, `testSearchForOffsetWithSize2`, etc and leave a comment explaining the scenario. I really don't like super long test names. Like @kirktrue said, you end up just not reading them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18989 Optimize FileRecord#searchForOffsetWithSize [kafka]

Posted by "m1a2st (via GitHub)" <gi...@apache.org>.
m1a2st commented on code in PR #19214:
URL: https://github.com/apache/kafka/pull/19214#discussion_r2001205036


##########
clients/src/test/java/org/apache/kafka/common/record/FileRecordsTest.java:
##########
@@ -518,6 +523,176 @@ public void testBytesLengthOfWriteTo() throws IOException {
         verify(channel).transferFrom(any(), anyLong(), eq((long) size - firstWritten));
     }
 
+    /**
+     * Test two condition
+     * 1. If the target offset equals to the base offset of the first batch
+     * 2. If the target offset < the base offset of the first batch
+     */
+    @ParameterizedTest
+    @ValueSource(longs = {5, 10})
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchMatch(long baseOffset) throws IOException {
+        File mockFile = mock(File.class);
+        FileChannel mockChannel = mock(FileChannel.class);
+        FileLogInputStream.FileChannelRecordBatch batch = mock(FileLogInputStream.FileChannelRecordBatch.class);
+        when(batch.baseOffset()).thenReturn(baseOffset);
+
+        FileRecords fileRecords = Mockito.spy(new FileRecords(mockFile, mockChannel, 0, 100, false));
+        mockFileRecordBatches(fileRecords, batch);
+
+        FileRecords.LogOffsetPosition result = fileRecords.searchForOffsetWithSize(5L, 0);
+
+        assertEquals(FileRecords.LogOffsetPosition.fromBatch(batch), result);
+        verify(batch, never()).lastOffset();
+    }
+
+    @Test
+    public void testSearchForOffsetWithSizeLastOffsetCallCountFirstBatchLastOffsetMatch() throws IOException {

Review Comment:
   Good idea, addressed it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org