You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2021/01/13 12:32:38 UTC

[GitHub] [bookkeeper] hangc0276 opened a new pull request #2535: fix ledgerDirsMonitor's listener doesn't work well on dbLedgerStorage

hangc0276 opened a new pull request #2535:
URL: https://github.com/apache/bookkeeper/pull/2535


   ### Motivation
   For `LedgerDirsMonitor`, it init and start in bookie startup according to `ledgersDirsManager`, and add listeners during Ledger Storage initialization. However, for dbLedgerStorage initialization, it create another LedgerDirsManager(ldm) to init `SingleDirectoryDbLedgerStorage` instance. During `SingleDirectoryDbLedgerStorage` initialization, 
   ```Java
   for (File ledgerDir : ledgerDirsManager.getAllLedgerDirs()) {
               // Create a ledger dirs manager for the single directory
               File[] dirs = new File[1];
               // Remove the `/current` suffix which will be appended again by LedgersDirManager
               dirs[0] = ledgerDir.getParentFile();
               LedgerDirsManager ldm = new LedgerDirsManager(conf, dirs, ledgerDirsManager.getDiskChecker(), statsLogger);
               ledgerStorageList.add(newSingleDirectoryDbLedgerStorage(conf, ledgerManager, ldm, indexDirsManager,
                       stateManager, checkpointSource, checkpointer, statsLogger, gcExecutor, perDirectoryWriteCacheSize,
                       perDirectoryReadCacheSize));
               dirsManagers.add(ldm);
           }
   ```
   During `SingleDirectoryDbLedgerStorage` initialization, it create `EntryLogManagerForSingleEntryLog` instance and register individual listener to ledgerDirsManager which create in `dbLedgerStorage` initialization.
   ```Java
   EntryLogManagerForSingleEntryLog(ServerConfiguration conf, LedgerDirsManager ledgerDirsManager,
               EntryLoggerAllocator entryLoggerAllocator, List<EntryLogger.EntryLogListener> listeners,
               EntryLogger.RecentEntryLogsStatus recentlyCreatedEntryLogsStatus) {
           super(conf, ledgerDirsManager, entryLoggerAllocator, listeners);
           this.rotatedLogChannels = new LinkedList<BufferedLogChannel>();
           this.recentlyCreatedEntryLogsStatus = recentlyCreatedEntryLogsStatus;
           // Register listener for disk full notifications.
           ledgerDirsManager.addLedgerDirsListener(getLedgerDirsListener());
       }
   
       private LedgerDirsListener getLedgerDirsListener() {
           return new LedgerDirsListener() {
               @Override
               public void diskFull(File disk) {
                   // If the current entry log disk is full, then create new
                   // entry log.
                   BufferedLogChannel currentActiveLogChannel = activeLogChannel;
                   if (currentActiveLogChannel != null
                           && currentActiveLogChannel.getLogFile().getParentFile().equals(disk)) {
                       shouldCreateNewEntryLog.set(true);
                   }
               }
   
               @Override
               public void diskAlmostFull(File disk) {
                   // If the current entry log disk is almost full, then create new entry
                   // log.
                   BufferedLogChannel currentActiveLogChannel = activeLogChannel;
                   if (currentActiveLogChannel != null
                           && currentActiveLogChannel.getLogFile().getParentFile().equals(disk)) {
                       shouldCreateNewEntryLog.set(true);
                   }
               }
           };
       }
   ```
   
   However, the ledgerDirsMonitor, which initiate and start in bookie, lost control for ledgerDirsManagers which created in dbLedgerStorage initialization and the listener registered in `EntryLogManagerForSingleEntryLog` will never be triggered.
   
   ### Changes
   1. create individual ledgerDirsMonitor instance in dbLedgerStorage to control newly created ledgerDirsManagers (ldm).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [bookkeeper] dlg99 commented on pull request #2535: fix ledgerDirsMonitor's listener doesn't work well on dbLedgerStorage

Posted by GitBox <gi...@apache.org>.
dlg99 commented on pull request #2535:
URL: https://github.com/apache/bookkeeper/pull/2535#issuecomment-831373454


   @hangc0276 can you please rebase/resolve conflicts and address failing CI checks?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [bookkeeper] eolivelli commented on pull request #2535: fix ledgerDirsMonitor's listener doesn't work well on dbLedgerStorage

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #2535:
URL: https://github.com/apache/bookkeeper/pull/2535#issuecomment-759421472


   @hangc0276 can you please check if we already have tests that cover this change ?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [bookkeeper] hangc0276 commented on pull request #2535: fix ledgerDirsMonitor's listener doesn't work well on dbLedgerStorage

Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #2535:
URL: https://github.com/apache/bookkeeper/pull/2535#issuecomment-901172115


   > @hangc0276 can you please resolve the conflicts ?
   
   @eolivelli Ok, I will resolve it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [bookkeeper] eolivelli commented on pull request #2535: fix ledgerDirsMonitor's listener doesn't work well on dbLedgerStorage

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #2535:
URL: https://github.com/apache/bookkeeper/pull/2535#issuecomment-768098342


   @sijie @merlimat @dlg99 please take a look


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [bookkeeper] eolivelli commented on pull request #2535: fix ledgerDirsMonitor's listener doesn't work well on dbLedgerStorage

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #2535:
URL: https://github.com/apache/bookkeeper/pull/2535#issuecomment-901059789


   @hangc0276 can you please resolve the conflicts ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [bookkeeper] eolivelli commented on pull request #2535: fix ledgerDirsMonitor's listener doesn't work well on dbLedgerStorage

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #2535:
URL: https://github.com/apache/bookkeeper/pull/2535#issuecomment-901059789


   @hangc0276 can you please resolve the conflicts ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [bookkeeper] dlg99 commented on pull request #2535: fix ledgerDirsMonitor's listener doesn't work well on dbLedgerStorage

Posted by GitBox <gi...@apache.org>.
dlg99 commented on pull request #2535:
URL: https://github.com/apache/bookkeeper/pull/2535#issuecomment-768423900


   LGTM but needs a test that listeners registered in `EntryLogManagerForSingleEntryLog` trigger as expected


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [bookkeeper] hangc0276 commented on pull request #2535: fix ledgerDirsMonitor's listener doesn't work well on dbLedgerStorage

Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #2535:
URL: https://github.com/apache/bookkeeper/pull/2535#issuecomment-770818293


   > @hangc0276 can you please address @dlg99's comment and add a test ?
   > also PR Validation failed, can you please take a look ?
   
   OK,I will add the test.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [bookkeeper] eolivelli commented on pull request #2535: fix ledgerDirsMonitor's listener doesn't work well on dbLedgerStorage

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #2535:
URL: https://github.com/apache/bookkeeper/pull/2535#issuecomment-770730373


   @hangc0276  can you please address @dlg99's comment and add a test ?
   also PR Validation failed, can you please take a look ?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [bookkeeper] hangc0276 commented on pull request #2535: fix ledgerDirsMonitor's listener doesn't work well on dbLedgerStorage

Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #2535:
URL: https://github.com/apache/bookkeeper/pull/2535#issuecomment-901172115


   > @hangc0276 can you please resolve the conflicts ?
   
   @eolivelli Ok, I will resolve it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org