[GitHub] [pulsar] pheecian commented on a change in pull request #6717: fix filesystem offload oom based on https://github.com/apache/pulsar/…

[GitBox <gi...@apache.org>]

pheecian commented on a change in pull request #6717: fix filesystem offload oom based on https://github.com/apache/pulsar/…
URL: https://github.com/apache/pulsar/pull/6717#discussion_r407047545
 
 

 ##########
 File path: tiered-storage/file-system/src/main/java/org/apache/bookkeeper/mledger/offload/filesystem/impl/FileSystemManagedLedgerOffloader.java
 ##########
 @@ -188,12 +191,15 @@ public void run() {
                 AtomicLong haveOffloadEntryNumber = new AtomicLong(0);
                 long needToOffloadFirstEntryNumber = 0;
                 CountDownLatch countDownLatch;
+                //avoid prefetch too much data into memory
+                ArrayBlockingQueue<Boolean> tasks = new ArrayBlockingQueue<>(PREFETCH_ROUNDS);
                 do {
                     long end = Math.min(needToOffloadFirstEntryNumber + ENTRIES_PER_READ - 1, readHandle.getLastAddConfirmed());
                     log.debug("read ledger entries. start: {}, end: {}", needToOffloadFirstEntryNumber, end);
                     LedgerEntries ledgerEntriesOnce = readHandle.readAsync(needToOffloadFirstEntryNumber, end).get();
 
 Review comment:
   the blockingQueue is to block the whole do while loop(to be precisely, the read and submit) Here, not try to limit fileSystemWriter. I understand that the FileSystemWriter will be executed orderly. My intention is to limit the whole do while loop because the do while loop((readAsync().get()) read data from ledger in advance too much too quickly(and lead to consume much memory), more than  FileSystemWriter Threads can handle timely. So read in advance, or put in other words, prefetch, should be controlled to some degree. So I introduce a blocking queue and hence a producer-consumer mock-up. the do while loop(read and submit) acts like a producer, the FileSystemWriter acts as a consumer. @congbobo184 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services