You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/06/09 16:22:04 UTC
[GitHub] [pulsar] ericwangdka opened a new issue #7214: bookie server's log : No ledger found while reading entry
ericwangdka opened a new issue #7214:
URL: https://github.com/apache/pulsar/issues/7214
it shows every 5min ，it appears reduce the throughtout how can i to do with it
11:22:16.266 [pool-11-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file /data10/bk-data/current/4429.log for logId 17449.
11:22:16.326 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog - Synced entry logger 17447 to disk.
11:22:16.350 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.Journal - garbage collected journal 172315ff9fb.txn
11:23:03.552 [BookieJournal-3181] INFO org.apache.bookkeeper.bookie.JournalChannel - Opening journal /data2/bk-journal/current/172315ffad3.txn
11:23:11.041 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 0 from ledger: 1500467
11:23:11.041 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 1492 from ledger: 1500467
11:23:11.044 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 702 from ledger: 1500467
11:23:11.045 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 705 from ledger: 1500467
11:23:11.046 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 708 from ledger: 1500467
11:23:11.047 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 711 from ledger: 1500467
11:23:11.048 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 714 from ledger: 1500467
11:23:11.048 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 717 from ledger: 1500467
11:23:11.048 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 720 from ledger: 1500467
11:23:11.048 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 723 from ledger: 1500467
11:23:11.049 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 726 from ledger: 1500467
11:23:11.049 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 729 from ledger: 1500467
11:23:11.049 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 732 from ledger: 1500467
11:23:11.049 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 735 from ledger: 1500467
11:23:11.049 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 738 from ledger: 1500467
11:23:11.050 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 741 from ledger: 1500467
11:23:11.050 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 744 from ledger: 1500467
11:23:11.050 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 747 from ledger: 1500467
11:23:11.050 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 750 from ledger: 1500467
11:23:11.050 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 753 from ledger: 1500467
11:23:11.051 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 756 from ledger: 1500467
11:23:11.052 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 759 from ledger: 1500467
11:23:11.052 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 762 from ledger: 1500467
11:23:11.052 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 765 from ledger: 1500467
11:23:11.052 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 768 from ledger: 1500467
11:23:11.052 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 771 from ledger: 1500467
11:23:11.052 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 774 from ledger: 1500467
11:23:11.053 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 777 from ledger: 1500467
11:23:11.053 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 780 from ledger: 1500467
11:23:11.053 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 783 from ledger: 1500467
11:23:11.053 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 786 from ledger: 1500467
11:23:11.053 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 789 from ledger: 1500467
11:23:11.054 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 792 from ledger: 1500467
11:23:11.054 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 795 from ledger: 1500467
11:23:11.054 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 798 from ledger: 1500467
11:23:11.054 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 0 from ledger: 1500467
11:23:11.054 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 1492 from ledger: 1500467
11:23:12.829 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file : createNewLog = false, reachEntryLogLimit = true
11:23:12.829 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Flushing entry logger 17625 back to filesystem, pending for syncing entry loggers : [BufferedChannel{logId=17625, logFile=/data4/bk-data/current/44d9.log, ledgerIdAssigned=-1}].
11:23:12.831 [pool-5-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file /data4/bk-data/current/44db.log for logId 17627.
11:23:13.274 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog - Synced entry logger 17625 to disk.
11:23:15.557 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file : createNewLog = false, reachEntryLogLimit = true
11:23:15.557 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Flushing entry logger 17438 back to filesystem, pending for syncing entry loggers : [BufferedChannel{logId=17438, logFile=/data8/bk-data/current/441e.log, ledgerIdAssigned=-1}].
11:23:15.559 [pool-9-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file /data8/bk-data/current/4420.log for logId 17440.
11:23:16.595 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog - Synced entry logger 17438 to disk.
11:23:17.710 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.Journal - garbage collected journal 172315ffacd.txn
11:24:13.007 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file : createNewLog = false, reachEntryLogLimit = true
11:24:13.007 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Flushing entry logger 17498 back to filesystem, pending for syncing entry loggers : [BufferedChannel{logId=17498, logFile=/data6/bk-data/current/445a.log, ledgerIdAssigned=-1}].
11:24:13.013 [pool-7-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file /data6/bk-data/current/445c.log for logId 17500.
11:24:13.112 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog - Synced entry logger 17498 to disk.
11:24:20.822 [BookieJournal-3181] INFO org.apache.bookkeeper.bookie.JournalChannel - Opening journal /data3/bk-journal/current/172315ffa02.txn
11:25:15.197 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file : createNewLog = false, reachEntryLogLimit = true
11:25:15.197 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Flushing entry logger 17406 back to filesystem, pending for syncing entry loggers : [BufferedChannel{logId=17406, logFile=/data9/bk-data/current/43fe.log, ledgerIdAssigned=-1}].
11:25:15.198 [pool-10-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file /data9/bk-data/current/4400.log for logId 17408.
11:25:15.296 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog - Synced entry logger 17406 to disk.
11:25:15.455 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.Journal - garbage collected journal 172315ff9fc.txn
11:26:51.993 [BookieJournal-3181] INFO org.apache.bookkeeper.bookie.JournalChannel - Opening journal /data2/bk-journal/current/172315ffad4.txn
11:27:16.631 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.Journal - garbage collected journal 172315fface.txn
11:28:04.343 [BookieJournal-3181] INFO org.apache.bookkeeper.bookie.JournalChannel - Opening journal /data3/bk-journal/current/172315ffa03.txn
11:28:11.056 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 0 from ledger: 1500467
11:28:11.056 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 1492 from ledger: 1500467
11:28:11.058 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 702 from ledger: 1500467
11:28:11.059 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 705 from ledger: 1500467
11:28:11.060 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 708 from ledger: 1500467
11:28:11.061 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 711 from ledger: 1500467
11:28:11.062 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 714 from ledger: 1500467
11:28:11.064 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 717 from ledger: 1500467
11:28:11.064 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 720 from ledger: 1500467
11:28:11.065 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 723 from ledger: 1500467
11:28:11.065 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 726 from ledger: 1500467
11:28:11.065 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 729 from ledger: 1500467
11:28:11.065 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 732 from ledger: 1500467
11:28:11.065 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 735 from ledger: 1500467
11:28:11.065 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 738 from ledger: 1500467
11:28:11.066 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 741 from ledger: 1500467
11:28:11.066 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 744 from ledger: 1500467
11:28:11.066 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 747 from ledger: 1500467
11:28:11.066 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 750 from ledger: 1500467
11:28:11.066 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 753 from ledger: 1500467
11:28:11.066 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 756 from ledger: 1500467
11:28:11.067 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 759 from ledger: 1500467
11:28:11.067 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 762 from ledger: 1500467
11:28:11.067 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 765 from ledger: 1500467
11:28:11.067 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 768 from ledger: 1500467
11:28:11.067 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 771 from ledger: 1500467
11:28:11.067 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 774 from ledger: 1500467
11:28:11.068 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 777 from ledger: 1500467
11:28:11.068 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 780 from ledger: 1500467
11:28:11.068 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 783 from ledger: 1500467
11:28:11.068 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 786 from ledger: 1500467
11:28:11.068 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 789 from ledger: 1500467
11:28:11.069 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 792 from ledger: 1500467
11:28:11.069 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 795 from ledger: 1500467
11:28:11.069 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 798 from ledger: 1500467
11:28:11.069 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 0 from ledger: 1500467
11:28:11.069 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 1492 from ledger: 1500467
11:28:13.499 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file : createNewLog = false, reachEntryLogLimit = true
11:28:13.499 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Flushing entry logger 17515 back to filesystem, pending for syncing entry loggers : [BufferedChannel{logId=17515, logFile=/data5/bk-data/current/446b.log, ledgerIdAssigned=-1}].
11:28:13.500 [pool-6-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file /data5/bk-data/current/446d.log for logId 17517.
11:28:15.109 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog - Synced entry logger 17515 to disk.
11:28:15.327 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file : createNewLog = false, reachEntryLogLimit = true
11:28:15.328 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Flushing entry logger 17235 back to filesystem, pending for syncing entry loggers : [BufferedChannel{logId=17235, logFile=/data7/bk-data/current/4353.log, ledgerIdAssigned=-1}].
11:28:15.328 [pool-8-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file /data7/bk-data/current/4355.log for logId 17237.
11:28:15.732 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog - Synced entry logger 17235 to disk.
11:28:15.940 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file : createNewLog = false, reachEntryLogLimit = true
11:28:15.940 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Flushing entry logger 17439 back to filesystem, pending for syncing entry loggers : [BufferedChannel{logId=17439, logFile=/data8/bk-data/current/441f.log, ledgerIdAssigned=-1}].
11:28:15.940 [pool-9-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file /data8/bk-data/current/4421.log for logId 17441.
11:28:16.203 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog - Synced entry logger 17439 to disk.
11:28:16.963 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.Journal - garbage collected journal 172315ff9fd.txn
11:30:05.028 [BookieJournal-3181] INFO org.apache.bookkeeper.bookie.JournalChannel - Opening journal /data2/bk-journal/current/172315ffad5.txn
11:30:15.411 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.Journal - garbage collected journal 172315ffacf.txn
11:31:12.643 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file : createNewLog = false, reachEntryLogLimit = true
11:31:12.643 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Flushing entry logger 17626 back to filesystem, pending for syncing entry loggers : [BufferedChannel{logId=17626, logFile=/data4/bk-data/current/44da.log, ledgerIdAssigned=-1}].
11:31:12.644 [pool-5-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file /data4/bk-data/current/44dc.log for logId 17628.
11:31:12.676 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog - Synced entry logger 17626 to disk.
11:32:00.723 [BookieJournal-3181] INFO org.apache.bookkeeper.bookie.JournalChannel - Opening journal /data3/bk-journal/current/172315ffa04.txn
11:32:15.043 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file : createNewLog = false, reachEntryLogLimit = true
11:32:15.044 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Flushing entry logger 17407 back to filesystem, pending for syncing entry loggers : [BufferedChannel{logId=17407, logFile=/data9/bk-data/current/43ff.log, ledgerIdAssigned=-1}].
11:32:15.045 [pool-10-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file /data9/bk-data/current/4401.log for logId 17409.
11:32:15.356 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog - Synced entry logger 17407 to disk.
11:32:15.508 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.Journal - garbage collected journal 172315ff9fe.txn
11:33:11.069 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 0 from ledger: 1500467
11:33:11.069 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 1492 from ledger: 1500467
11:33:11.072 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 702 from ledger: 1500467
11:33:11.073 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 705 from ledger: 1500467
11:33:11.074 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 708 from ledger: 1500467
11:33:11.075 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 711 from ledger: 1500467
11:33:11.076 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 714 from ledger: 1500467
11:33:11.076 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 717 from ledger: 1500467
11:33:11.076 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 720 from ledger: 1500467
11:33:11.077 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 723 from ledger: 1500467
11:33:11.077 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 726 from ledger: 1500467
11:33:11.077 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 729 from ledger: 1500467
11:33:11.077 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 732 from ledger: 1500467
11:33:11.077 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 735 from ledger: 1500467
11:33:11.078 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 738 from ledger: 1500467
11:33:11.078 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 741 from ledger: 1500467
11:33:11.078 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 744 from ledger: 1500467
11:33:11.078 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 747 from ledger: 1500467
11:33:11.078 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 750 from ledger: 1500467
11:33:11.078 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 753 from ledger: 1500467
11:33:11.080 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 756 from ledger: 1500467
11:33:11.080 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 759 from ledger: 1500467
11:33:11.080 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 762 from ledger: 1500467
11:33:11.080 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 765 from ledger: 1500467
11:33:11.080 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 768 from ledger: 1500467
11:33:11.080 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 771 from ledger: 1500467
11:33:11.081 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 774 from ledger: 1500467
11:33:11.081 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 777 from ledger: 1500467
11:33:11.081 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 780 from ledger: 1500467
11:33:11.081 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 783 from ledger: 1500467
11:33:11.081 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 786 from ledger: 1500467
11:33:11.082 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 789 from ledger: 1500467
11:33:11.082 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 792 from ledger: 1500467
11:33:11.082 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 795 from ledger: 1500467
11:33:11.082 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 798 from ledger: 1500467
11:33:11.082 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 0 from ledger: 1500467
11:33:11.082 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 1492 from ledger: 1500467
11:33:12.922 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file : createNewLog = false, reachEntryLogLimit = true
11:33:12.922 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Flushing entry logger 17516 back to filesystem, pending for syncing entry loggers : [BufferedChannel{logId=17516, logFile=/data5/bk-data/current/446c.log, ledgerIdAssigned=-1}].
11:33:12.923 [pool-6-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file /data5/bk-data/current/446e.log for logId 17518.
11:33:13.542 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog - Synced entry logger 17516 to disk.
11:33:13.871 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file : createNewLog = false, reachEntryLogLimit = true
11:33:13.871 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Flushing entry logger 17236 back to filesystem, pending for syncing entry loggers : [BufferedChannel{logId=17236, logFile=/data7/bk-data/current/4354.log, ledgerIdAssigned=-1}].
11:33:13.872 [pool-8-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file /data7/bk-data/current/4356.log for logId 17238.
11:33:14.532 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog - Synced entry logger 17236 to disk.
11:33:51.548 [BookieJournal-3181] INFO org.apache.bookkeeper.bookie.JournalChannel - Opening journal /data2/bk-journal/current/172315ffad6.txn
11:34:12.723 [GarbageCollectorThread-15-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17513
11:34:12.723 [GarbageCollectorThread-15-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17514
11:34:12.723 [GarbageCollectorThread-15-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17515
11:34:12.723 [GarbageCollectorThread-15-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17516
11:34:12.726 [GarbageCollectorThread-21-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17234
11:34:12.726 [GarbageCollectorThread-21-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17235
11:34:12.726 [GarbageCollectorThread-21-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17236
11:34:12.727 [GarbageCollectorThread-24-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17437
11:34:12.727 [GarbageCollectorThread-24-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17438
11:34:12.727 [GarbageCollectorThread-24-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17439
11:34:12.727 [GarbageCollectorThread-12-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17622
11:34:12.727 [GarbageCollectorThread-12-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17623
11:34:12.727 [GarbageCollectorThread-12-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17624
11:34:12.728 [GarbageCollectorThread-12-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17625
11:34:12.728 [GarbageCollectorThread-12-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17626
11:34:12.728 [GarbageCollectorThread-30-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17446
11:34:12.728 [GarbageCollectorThread-30-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17447
11:34:12.730 [GarbageCollectorThread-18-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17496
11:34:12.730 [GarbageCollectorThread-18-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17497
11:34:12.730 [GarbageCollectorThread-18-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17498
11:34:12.731 [GarbageCollectorThread-27-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17404
11:34:12.731 [GarbageCollectorThread-27-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17405
11:34:12.731 [GarbageCollectorThread-27-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17406
11:34:12.731 [GarbageCollectorThread-27-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 17407
11:34:12.801 [GarbageCollectorThread-30-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Deleting entryLogId 17443 as it has no active ledgers!
11:34:13.028 [GarbageCollectorThread-30-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Removing entry log metadata for 17443
11:34:13.113 [db-storage-cleanup-14-1] INFO org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleting indexes for ledgers: [2080926, 2080940, 2080905, 2080807, 2080933, 2080821, 2080849]
11:34:13.121 [db-storage-cleanup-14-1] INFO org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleted indexes for 343 entries from 7 ledgers in 0.008 seconds
11:34:13.170 [db-storage-cleanup-17-1] INFO org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleting indexes for ledgers: [2080941, 2080892, 2080794, 2080969, 2080808, 2085750, 2080836]
11:34:13.341 [db-storage-cleanup-17-1] INFO org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleted indexes for 274650 entries from 7 ledgers in 0.171 seconds
11:34:13.468 [db-storage-cleanup-20-1] INFO org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleting indexes for ledgers: [2080844, 2065276, 2080907, 2080809, 2080921, 2085751, 2080837, 2066676, 2080851, 2080802, 2080914, 2080977]
11:34:13.523 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file : createNewLog = false, reachEntryLogLimit = true
11:34:13.523 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Flushing entry logger 17440 back to filesystem, pending for syncing entry loggers : [BufferedChannel{logId=17440, logFile=/data8/bk-data/current/4420.log, ledgerIdAssigned=-1}].
11:34:13.526 [pool-9-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file /data8/bk-data/current/4422.log for logId 17442.
11:34:13.601 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog - Synced entry logger 17440 to disk.
11:34:13.602 [db-storage-cleanup-23-1] INFO org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleting indexes for ledgers: [2080957, 2080922, 2085752, 2066677, 2086004, 2080915, 2085682, 2080943, 2080845, 2080971, 2080810, 2080999, 2065221, 2080964, 2080803]
11:34:13.809 [db-storage-cleanup-26-1] INFO org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleting indexes for ledgers: [2080797, 2080923, 2081000, 2080839, 2080902, 2080965, 2080944]
11:34:13.810 [db-storage-cleanup-26-1] INFO org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleted indexes for 335 entries from 7 ledgers in 0.0 seconds
11:34:13.856 [db-storage-cleanup-20-1] INFO org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleted indexes for 241583 entries from 12 ledgers in 0.388 seconds
11:34:13.959 [db-storage-cleanup-29-1] INFO org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleting indexes for ledgers: [2080910, 2080973, 2085740, 2080812, 2079496, 2085432, 2080805, 2086580, 2080819]
11:34:13.968 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.Journal - garbage collected journal 172315ffad0.txn
11:34:14.184 [db-storage-cleanup-23-1] INFO org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleted indexes for 956660 entries from 15 ledgers in 0.582 seconds
11:34:14.845 [db-storage-cleanup-29-1] INFO org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleted indexes for 1012185 entries from 9 ledgers in 0.886 seconds
11:34:47.825 [BookieJournal-3181] INFO org.apache.bookkeeper.bookie.JournalChannel - Opening journal /data3/bk-journal/current/172315ffa05.txn
11:35:12.818 [db-storage-cleanup-11-1] INFO org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleting indexes for ledgers: [2085951, 2080890, 2080953, 2080848, 2073680]
11:35:13.346 [db-storage-cleanup-11-1] INFO org.apache.bookkeeper.bookie.storage.ldb.EntryLocationIndex - Deleted indexes for 466272 entries from 5 ledgers in 0.527 seconds
11:35:18.127 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.Journal - garbage collected journal 172315ff9ff.txn
11:37:15.523 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file : createNewLog = false, reachEntryLogLimit = true
11:37:15.523 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Flushing entry logger 17408 back to filesystem, pending for syncing entry loggers : [BufferedChannel{logId=17408, logFile=/data9/bk-data/current/4400.log, ledgerIdAssigned=-1}].
11:37:15.527 [pool-10-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file /data9/bk-data/current/4402.log for logId 17410.
11:37:15.883 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog - Synced entry logger 17408 to disk.
11:37:16.520 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file : createNewLog = false, reachEntryLogLimit = true
11:37:16.520 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Flushing entry logger 17448 back to filesystem, pending for syncing entry loggers : [BufferedChannel{logId=17448, logFile=/data10/bk-data/current/4428.log, ledgerIdAssigned=-1}].
11:37:16.520 [pool-11-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file /data10/bk-data/current/442a.log for logId 17450.
11:37:16.566 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog - Synced entry logger 17448 to disk.
11:37:17.957 [BookieJournal-3181] INFO org.apache.bookkeeper.bookie.JournalChannel - Opening journal /data2/bk-journal/current/172315ffad7.txn
11:38:11.083 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 0 from ledger: 1500467
11:38:11.083 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 1492 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 702 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 705 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 708 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 711 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 714 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 717 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 720 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 723 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 726 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 729 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 732 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 735 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 738 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 741 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 744 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 747 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 750 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 753 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 756 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 759 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 762 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 765 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 768 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 771 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 774 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 777 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 780 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 783 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 786 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 789 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 792 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 795 from ledger: 1500467
11:38:11.199 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 798 from ledger: 1500467
11:38:11.200 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 0 from ledger: 1500467
11:38:11.200 [BookieReadThreadPool-OrderedExecutor-19-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 1492 from ledger: 1500467
11:38:12.672 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file : createNewLog = false, reachEntryLogLimit = true
11:38:12.672 [SyncThread-8-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Flushing entry logger 17627 back to filesystem, pending for syncing entry loggers : [BufferedChannel{logId=17627, logFile=/data4/bk-data/current/44db.log, ledgerIdAssigned=-1}].
11:38:12.674 [pool-5-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file /data4/bk-data/current/44dd.log for logId 17629.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7214: bookie server's log : No ledger found while reading entry
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7214:
URL: https://github.com/apache/pulsar/issues/7214#issuecomment-642145544
@ericwangdka it looks like the entry was lost. Did you replace disks before or did you do any operations related to bookies before?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ericwangdka commented on issue #7214: bookie server's log : No ledger found while reading entry
Posted by GitBox <gi...@apache.org>.
ericwangdka commented on issue #7214:
URL: https://github.com/apache/pulsar/issues/7214#issuecomment-643754849
@codelipenghui it appears the ledger could be see in zookeeper ,however it cound not be found in bookies ,how can i to do deal with it
.............................................bookeeper.conf#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
## Bookie settings
#############################################################################
## Server parameters
#############################################################################
# Port that bookie server listen on
bookiePort=3181
# Directories BookKeeper outputs its write ahead log.
# Could define multi directories to store write head logs, separated by ','.
# For example:
# journalDirectories=/tmp/bk-journal1,/tmp/bk-journal2
# If journalDirectories is set, bookies will skip journalDirectory and use
# this setting directory.
# journalDirectories=/tmp/bk-journal
journalDirectories=/data2/bk-journal,/data3/bk-journal
# Directory Bookkeeper outputs its write ahead log
# @deprecated since 4.5.0. journalDirectories is preferred over journalDirectory.
#journalDirectory=data/bookkeeper/journal
# Configure the bookie to allow/disallow multiple ledger/index/journal directories
# in the same filesystem disk partition
# allowMultipleDirsUnderSameDiskPartition=false
# Minimum safe Usable size to be available in index directory for bookie to create
# Index File while replaying journal at the time of bookie Start in Readonly Mode (in bytes)
minUsableSizeForIndexFileCreation=1073741824
# Set the network interface that the bookie should listen on.
# If not set, the bookie will listen on all interfaces.
# listeningInterface=eth0
# Configure a specific hostname or IP address that the bookie should use to advertise itself to
# clients. If not set, bookie will advertised its own IP address or hostname, depending on the
# listeningInterface and useHostNameAsBookieID settings.
#advertisedAddress=
advertisedAddress=132.90.117.238
# Whether the bookie allowed to use a loopback interface as its primary
# interface(i.e. the interface it uses to establish its identity)?
# By default, loopback interfaces are not allowed as the primary
# interface.
# Using a loopback interface as the primary interface usually indicates
# a configuration error. For example, its fairly common in some VPS setups
# to not configure a hostname, or to have the hostname resolve to
# 127.0.0.1. If this is the case, then all bookies in the cluster will
# establish their identities as 127.0.0.1:3181, and only one will be able
# to join the cluster. For VPSs configured like this, you should explicitly
# set the listening interface.
allowLoopback=false
# Interval to watch whether bookie is dead or not, in milliseconds
bookieDeathWatchInterval=1000
# When entryLogPerLedgerEnabled is enabled, checkpoint doesn't happens
# when a new active entrylog is created / previous one is rolled over.
# Instead SyncThread checkpoints periodically with 'flushInterval' delay
# (in milliseconds) in between executions. Checkpoint flushes both ledger
# entryLogs and ledger index pages to disk.
# Flushing entrylog and index files will introduce much random disk I/O.
# If separating journal dir and ledger dirs each on different devices,
# flushing would not affect performance. But if putting journal dir
# and ledger dirs on same device, performance degrade significantly
# on too frequent flushing. You can consider increment flush interval
# to get better performance, but you need to pay more time on bookie
# server restart after failure.
# This config is used only when entryLogPerLedgerEnabled is enabled.
flushInterval=60000
# Allow the expansion of bookie storage capacity. Newly added ledger
# and index dirs must be empty.
# allowStorageExpansion=false
# Whether the bookie should use its hostname to register with the
# co-ordination service(eg: Zookeeper service).
# When false, bookie will use its ipaddress for the registration.
# Defaults to false.
useHostNameAsBookieID=false
# Whether the bookie is allowed to use an ephemeral port (port 0) as its
# server port. By default, an ephemeral port is not allowed.
# Using an ephemeral port as the service port usually indicates a configuration
# error. However, in unit tests, using an ephemeral port will address port
# conflict problems and allow running tests in parallel.
# allowEphemeralPorts=false
# Whether allow the bookie to listen for BookKeeper clients executed on the local JVM.
# enableLocalTransport=false
# Whether allow the bookie to disable bind on network interfaces,
# this bookie will be available only to BookKeeper clients executed on the local JVM.
# disableServerSocketBind=false
# The number of bytes we should use as chunk allocation for
# org.apache.bookkeeper.bookie.SkipListArena
# skipListArenaChunkSize=4194304
# The max size we should allocate from the skiplist arena. Allocations
# larger than this should be allocated directly by the VM to avoid fragmentation.
# skipListArenaMaxAllocSize=131072
# The bookie authentication provider factory class name.
# If this is null, no authentication will take place.
# bookieAuthProviderFactoryClass=null
#############################################################################
## Garbage collection settings
#############################################################################
# How long the interval to trigger next garbage collection, in milliseconds
# Since garbage collection is running in background, too frequent gc
# will heart performance. It is better to give a higher number of gc
# interval if there is enough disk capacity.
gcWaitTime=900000
# How long the interval to trigger next garbage collection of overreplicated
# ledgers, in milliseconds [Default: 1 day]. This should not be run very frequently
# since we read the metadata for all the ledgers on the bookie from zk
gcOverreplicatedLedgerWaitTime=86400000
# Number of threads that should handle write requests. if zero, the writes would
# be handled by netty threads directly.
#numAddWorkerThreads=0
numAddWorkerThreads=32
# Number of threads that should handle read requests. if zero, the reads would
# be handled by netty threads directly.
#numReadWorkerThreads=8
numReadWorkerThreads=32
# Number of threads that should be used for high priority requests
# (i.e. recovery reads and adds, and fencing).
numHighPriorityWorkerThreads=8
# If read workers threads are enabled, limit the number of pending requests, to
# avoid the executor queue to grow indefinitely
#maxPendingReadRequestsPerThread=2500
maxPendingReadRequestsPerThread=10000
# If add workers threads are enabled, limit the number of pending requests, to
# avoid the executor queue to grow indefinitely
maxPendingAddRequestsPerThread=10000
# Whether force compaction is allowed when the disk is full or almost full.
# Forcing GC may get some space back, but may also fill up disk space more quickly.
# This is because new log files are created before GC, while old garbage
# log files are deleted after GC.
# isForceGCAllowWhenNoSpace=false
# True if the bookie should double check readMetadata prior to gc
# verifyMetadataOnGC=false
#############################################################################
## TLS settings
#############################################################################
# TLS Provider (JDK or OpenSSL).
# tlsProvider=OpenSSL
# The path to the class that provides security.
# tlsProviderFactoryClass=org.apache.bookkeeper.security.SSLContextFactory
# Type of security used by server.
# tlsClientAuthentication=true
# Bookie Keystore type.
# tlsKeyStoreType=JKS
# Bookie Keystore location (path).
# tlsKeyStore=null
# Bookie Keystore password path, if the keystore is protected by a password.
# tlsKeyStorePasswordPath=null
# Bookie Truststore type.
# tlsTrustStoreType=null
# Bookie Truststore location (path).
# tlsTrustStore=null
# Bookie Truststore password path, if the trust store is protected by a password.
# tlsTrustStorePasswordPath=null
#############################################################################
## Long poll request parameter settings
#############################################################################
# The number of threads that should handle long poll requests.
# numLongPollWorkerThreads=10
# The tick duration in milliseconds for long poll requests.
# requestTimerTickDurationMs=10
# The number of ticks per wheel for the long poll request timer.
# requestTimerNumTicks=1024
#############################################################################
## AutoRecovery settings
#############################################################################
# The interval between auditor bookie checks.
# The auditor bookie check, checks ledger metadata to see which bookies should
# contain entries for each ledger. If a bookie which should contain entries is
# unavailable, then the ledger containing that entry is marked for recovery.
# Setting this to 0 disabled the periodic check. Bookie checks will still
# run when a bookie fails.
# The interval is specified in seconds.
auditorPeriodicBookieCheckInterval=86400
# The number of entries that a replication will rereplicate in parallel.
rereplicationEntryBatchSize=100
# Auto-replication
# The grace period, in seconds, that the replication worker waits before fencing and
# replicating a ledger fragment that's still being written to upon bookie failure.
openLedgerRereplicationGracePeriod=30
# Whether the bookie itself can start auto-recovery service also or not
autoRecoveryDaemonEnabled=true
# How long to wait, in seconds, before starting auto recovery of a lost bookie
lostBookieRecoveryDelay=0
#############################################################################
## Netty server settings
#############################################################################
# This settings is used to enabled/disabled Nagle's algorithm, which is a means of
# improving the efficiency of TCP/IP networks by reducing the number of packets
# that need to be sent over the network.
# If you are sending many small messages, such that more than one can fit in
# a single IP packet, setting server.tcpnodelay to false to enable Nagle algorithm
# can provide better performance.
# Default value is true.
serverTcpNoDelay=true
# This setting is used to send keep-alive messages on connection-oriented sockets.
# serverSockKeepalive=true
# The socket linger timeout on close.
# When enabled, a close or shutdown will not return until all queued messages for
# the socket have been successfully sent or the linger timeout has been reached.
# Otherwise, the call returns immediately and the closing is done in the background.
# serverTcpLinger=0
# The Recv ByteBuf allocator initial buf size.
# byteBufAllocatorSizeInitial=65536
# The Recv ByteBuf allocator min buf size.
# byteBufAllocatorSizeMin=65536
# The Recv ByteBuf allocator max buf size.
# byteBufAllocatorSizeMax=1048576
#############################################################################
## Journal settings
#############################################################################
# The journal format version to write.
# Available formats are 1-6:
# 1: no header
# 2: a header section was added
# 3: ledger key was introduced
# 4: fencing key was introduced
# 5: expanding header to 512 and padding writes to align sector size configured by `journalAlignmentSize`
# 6: persisting explicitLac is introduced
# By default, it is `6`.
# If you'd like to disable persisting ExplicitLac, you can set this config to < `6` and also
# fileInfoFormatVersionToWrite should be set to 0. If there is mismatch then the serverconfig is considered invalid.
# You can disable `padding-writes` by setting journal version back to `4`. This feature is available in 4.5.0
# and onward versions.
journalFormatVersionToWrite=5
# Max file size of journal file, in mega bytes
# A new journal file will be created when the old one reaches the file size limitation
journalMaxSizeMB=2048
# Max number of old journal file to kept
# Keep a number of old journal files would help data recovery in specia case
journalMaxBackups=5
# How much space should we pre-allocate at a time in the journal.
journalPreAllocSizeMB=16
# Size of the write buffers used for the journal
journalWriteBufferSizeKB=64
# Should we remove pages from page cache after force write
journalRemoveFromPageCache=true
# Should the data be fsynced on journal before acknowledgment.
# By default, data sync is enabled to guarantee durability of writes.
# Beware: while disabling data sync in the Bookie journal might improve the bookie write performance, it will also
# introduce the possibility of data loss. With no sync, the journal entries are written in the OS page cache but
# not flushed to disk. In case of power failure, the affected bookie might lose the unflushed data. If the ledger
# is replicated to multiple bookies, the chances of data loss are reduced though still present.
journalSyncData=true
# Should we group journal force writes, which optimize group commit
# for higher throughput
journalAdaptiveGroupWrites=true
# Maximum latency to impose on a journal write to achieve grouping
journalMaxGroupWaitMSec=1
# Maximum writes to buffer to achieve grouping
journalBufferedWritesThreshold=524288
# The number of threads that should handle journal callbacks
#numJournalCallbackThreads=8
numJournalCallbackThreads=40
# All the journal writes and commits should be aligned to given size.
# If not, zeros will be padded to align to given size.
# It only takes effects when journalFormatVersionToWrite is set to 5
journalAlignmentSize=4096
# Maximum entries to buffer to impose on a journal write to achieve grouping.
# journalBufferedEntriesThreshold=0
# If we should flush the journal when journal queue is empty
journalFlushWhenQueueEmpty=false
#############################################################################
## Ledger storage settings
#############################################################################
# Ledger storage implementation class
ledgerStorageClass=org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage
# Directory Bookkeeper outputs ledger snapshots
# could define multi directories to store snapshots, separated by ','
# For example:
# ledgerDirectories=/tmp/bk1-data,/tmp/bk2-data
#
# Ideally ledger dirs and journal dir are each in a differet device,
# which reduce the contention between random i/o and sequential write.
# It is possible to run with a single disk, but performance will be significantly lower.
# Directories to store index files. If not specified, will use ledgerDirectories to store.
#ledgerDirectories=data/bookkeeper/ledgers
ledgerDirectories=/data4/bk-data,/data5/bk-data,/data6/bk-data,/data7/bk-data,/data8/bk-data,/data9/bk-data,/data10/bk-data
# indexDirectories=data/bookkeeper/ledgers
# Interval at which the auditor will do a check of all ledgers in the cluster.
# By default this runs once a week. The interval is set in seconds.
# To disable the periodic check completely, set this to 0.
# Note that periodic checking will put extra load on the cluster, so it should
# not be run more frequently than once a day.
auditorPeriodicCheckInterval=604800
# Whether sorted-ledger storage enabled (default true)
# sortedLedgerStorageEnabled=ture
# The skip list data size limitation (default 64MB) in EntryMemTable
# skipListSizeLimit=67108864L
#############################################################################
## Ledger cache settings
#############################################################################
# Max number of ledger index files could be opened in bookie server
# If number of ledger index files reaches this limitation, bookie
# server started to swap some ledgers from memory to disk.
# Too frequent swap will affect performance. You can tune this number
# to gain performance according your requirements.
openFileLimit=0
# The fileinfo format version to write.
# Available formats are 0-1:
# 0: Initial version
# 1: persisting explicitLac is introduced
# By default, it is `1`.
# If you'd like to disable persisting ExplicitLac, you can set this config to 0 and
# also journalFormatVersionToWrite should be set to < 6. If there is mismatch then the
# serverconfig is considered invalid.
fileInfoFormatVersionToWrite=0
# Size of a index page in ledger cache, in bytes
# A larger index page can improve performance writing page to disk,
# which is efficent when you have small number of ledgers and these
# ledgers have similar number of entries.
# If you have large number of ledgers and each ledger has fewer entries,
# smaller index page would improve memory usage.
# pageSize=8192
# How many index pages provided in ledger cache
# If number of index pages reaches this limitation, bookie server
# starts to swap some ledgers from memory to disk. You can increment
# this value when you found swap became more frequent. But make sure
# pageLimit*pageSize should not more than JVM max memory limitation,
# otherwise you would got OutOfMemoryException.
# In general, incrementing pageLimit, using smaller index page would
# gain bettern performance in lager number of ledgers with fewer entries case
# If pageLimit is -1, bookie server will use 1/3 of JVM memory to compute
# the limitation of number of index pages.
pageLimit=0
#############################################################################
## Ledger manager settings
#############################################################################
# Ledger Manager Class
# What kind of ledger manager is used to manage how ledgers are stored, managed
# and garbage collected. Try to read 'BookKeeper Internals' for detail info.
# ledgerManagerFactoryClass=org.apache.bookkeeper.meta.HierarchicalLedgerManagerFactory
# @Drepcated - `ledgerManagerType` is deprecated in favor of using `ledgerManagerFactoryClass`.
# ledgerManagerType=hierarchical
# Root Zookeeper path to store ledger metadata
# This parameter is used by zookeeper-based ledger manager as a root znode to
# store all ledgers.
zkLedgersRootPath=/ledgers
#############################################################################
## Entry log settings
#############################################################################
# Max file size of entry logger, in bytes
# A new entry log file will be created when the old one reaches the file size limitation
logSizeLimit=1073741824
# Enable/Disable entry logger preallocation
entryLogFilePreallocationEnabled=true
# Entry log flush interval in bytes.
# Default is 0. 0 or less disables this feature and effectively flush
# happens on log rotation.
# Flushing in smaller chunks but more frequently reduces spikes in disk
# I/O. Flushing too frequently may also affect performance negatively.
# flushEntrylogBytes=0
# The number of bytes we should use as capacity for BufferedReadChannel. Default is 512 bytes.
readBufferSizeBytes=4096
# The number of bytes used as capacity for the write buffer. Default is 64KB.
writeBufferSizeBytes=65536
# Specifies if entryLog per ledger is enabled/disabled. If it is enabled, then there would be a
# active entrylog for each ledger. It would be ideal to enable this feature if the underlying
# storage device has multiple DiskPartitions or SSD and if in a given moment, entries of fewer
# number of active ledgers are written to a bookie.
# entryLogPerLedgerEnabled=false
#############################################################################
## Entry log compaction settings
#############################################################################
# Set the rate at which compaction will readd entries. The unit is adds per second.
compactionRate=1000
# If bookie is using hostname for registration and in ledger metadata then
# whether to use short hostname or FQDN hostname. Defaults to false.
# useShortHostName=false
# Threshold of minor compaction
# For those entry log files whose remaining size percentage reaches below
# this threshold will be compacted in a minor compaction.
# If it is set to less than zero, the minor compaction is disabled.
minorCompactionThreshold=0.2
# Interval to run minor compaction, in seconds
# If it is set to less than zero, the minor compaction is disabled.
# Note: should be greater than gcWaitTime.
minorCompactionInterval=3600
# Set the maximum number of entries which can be compacted without flushing.
# When compacting, the entries are written to the entrylog and the new offsets
# are cached in memory. Once the entrylog is flushed the index is updated with
# the new offsets. This parameter controls the number of entries added to the
# entrylog before a flush is forced. A higher value for this parameter means
# more memory will be used for offsets. Each offset consists of 3 longs.
# This parameter should _not_ be modified unless you know what you're doing.
# The default is 100,000.
compactionMaxOutstandingRequests=100000
# Threshold of major compaction
# For those entry log files whose remaining size percentage reaches below
# this threshold will be compacted in a major compaction.
# Those entry log files whose remaining size percentage is still
# higher than the threshold will never be compacted.
# If it is set to less than zero, the minor compaction is disabled.
majorCompactionThreshold=0.5
# Interval to run major compaction, in seconds
# If it is set to less than zero, the major compaction is disabled.
# Note: should be greater than gcWaitTime.
majorCompactionInterval=86400
# Throttle compaction by bytes or by entries.
isThrottleByBytes=false
# Set the rate at which compaction will readd entries. The unit is adds per second.
compactionRateByEntries=1000
# Set the rate at which compaction will readd entries. The unit is bytes added per second.
compactionRateByBytes=1000000
#############################################################################
## Statistics
#############################################################################
# Whether statistics are enabled
# enableStatistics=true
# Stats Provider Class (if statistics are enabled)
statsProviderClass=org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider
# Default port for Prometheus metrics exporter
prometheusStatsHttpPort=8000
#############################################################################
## Read-only mode support
#############################################################################
# If all ledger directories configured are full, then support only read requests for clients.
# If "readOnlyModeEnabled=true" then on all ledger disks full, bookie will be converted
# to read-only mode and serve only read requests. Otherwise the bookie will be shutdown.
# By default this will be disabled.
readOnlyModeEnabled=true
# Whether the bookie is force started in read only mode or not
# forceReadOnlyBookie=false
# Persiste the bookie status locally on the disks. So the bookies can keep their status upon restarts
# @Since 4.6
# persistBookieStatusEnabled=false
#############################################################################
## Disk utilization
#############################################################################
# For each ledger dir, maximum disk space which can be used.
# Default is 0.95f. i.e. 95% of disk can be used at most after which nothing will
# be written to that partition. If all ledger dir partions are full, then bookie
# will turn to readonly mode if 'readOnlyModeEnabled=true' is set, else it will
# shutdown.
# Valid values should be in between 0 and 1 (exclusive).
diskUsageThreshold=0.95
# The disk free space low water mark threshold.
# Disk is considered full when usage threshold is exceeded.
# Disk returns back to non-full state when usage is below low water mark threshold.
# This prevents it from going back and forth between these states frequently
# when concurrent writes and compaction are happening. This also prevent bookie from
# switching frequently between read-only and read-writes states in the same cases.
# diskUsageWarnThreshold=0.95
# Set the disk free space low water mark threshold. Disk is considered full when
# usage threshold is exceeded. Disk returns back to non-full state when usage is
# below low water mark threshold. This prevents it from going back and forth
# between these states frequently when concurrent writes and compaction are
# happening. This also prevent bookie from switching frequently between
# read-only and read-writes states in the same cases.
# diskUsageLwmThreshold=0.90
# Disk check interval in milli seconds, interval to check the ledger dirs usage.
# Default is 10000
diskCheckInterval=10000
#############################################################################
## ZooKeeper parameters
#############################################################################
# A list of one of more servers on which Zookeeper is running.
# The server list can be comma separated values, for example:
# zkServers=zk1:2181,zk2:2181,zk3:2181
#zkServers=localhost:2181
zkServers=132.90.117.238:2181,132.90.117.239:2181,132.90.117.240:2181,132.90.117.22:2181,132.90.117.32:2181
# ZooKeeper client session timeout in milliseconds
# Bookie server will exit if it received SESSION_EXPIRED because it
# was partitioned off from ZooKeeper for more than the session timeout
# JVM garbage collection, disk I/O will cause SESSION_EXPIRED.
# Increment this value could help avoiding this issue
zkTimeout=30000
# The Zookeeper client backoff retry start time in millis.
# zkRetryBackoffStartMs=1000
# The Zookeeper client backoff retry max time in millis.
# zkRetryBackoffMaxMs=10000
# Set ACLs on every node written on ZooKeeper, this way only allowed users
# will be able to read and write BookKeeper metadata stored on ZooKeeper.
# In order to make ACLs work you need to setup ZooKeeper JAAS authentication
# all the bookies and Client need to share the same user, and this is usually
# done using Kerberos authentication. See ZooKeeper documentation
zkEnableSecurity=false
#############################################################################
## Server parameters
#############################################################################
# The flag enables/disables starting the admin http server. Default value is 'false'.
httpServerEnabled=false
# The http server port to listen on. Default value is 8080.
# Use `8000` as the port to keep it consistent with prometheus stats provider
httpServerPort=8000
# The http server class
httpServerClass=org.apache.bookkeeper.http.vertx.VertxHttpServer
# Configure a list of server components to enable and load on a bookie server.
# This provides the plugin run extra services along with a bookie server.
#
# extraServerComponents=
#############################################################################
## DB Ledger storage configuration
#############################################################################
# These configs are used when the selected 'ledgerStorageClass' is
# org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage
# Size of Write Cache. Memory is allocated from JVM direct memory.
# Write cache is used to buffer entries before flushing into the entry log
# For good performance, it should be big enough to hold a substantial amount
# of entries in the flush interval
# By default it will be allocated to 1/4th of the available direct memory
dbStorage_writeCacheMaxSizeMb=
# Size of Read cache. Memory is allocated from JVM direct memory.
# This read cache is pre-filled doing read-ahead whenever a cache miss happens
# By default it will be allocated to 1/4th of the available direct memory
dbStorage_readAheadCacheMaxSizeMb=
# How many entries to pre-fill in cache after a read cache miss
#dbStorage_readAheadCacheBatchSize=1000
dbStorage_readAheadCacheBatchSize=5000
## RocksDB specific configurations
## DbLedgerStorage uses RocksDB to store the indexes from
## (ledgerId, entryId) -> (entryLog, offset)
# Size of RocksDB block-cache. For best performance, this cache
# should be big enough to hold a significant portion of the index
# database which can reach ~2GB in some cases
# Default is to use 10% of the direct memory size
dbStorage_rocksDB_blockCacheSize=
# Other RocksDB specific tunables
dbStorage_rocksDB_writeBufferSizeMB=64
dbStorage_rocksDB_sstSizeInMB=64
dbStorage_rocksDB_blockSize=65536
dbStorage_rocksDB_bloomFilterBitsPerKey=10
dbStorage_rocksDB_numLevels=-1
dbStorage_rocksDB_numFilesInLevel0=4
dbStorage_rocksDB_maxSizeInLevel1MB=256
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ericwangdka commented on issue #7214: bookie server's log : No ledger found while reading entry
Posted by GitBox <gi...@apache.org>.
ericwangdka commented on issue #7214:
URL: https://github.com/apache/pulsar/issues/7214#issuecomment-642145031
ledgerID: 1500467
LedgerMetadata{formatVersion=3, ensembleSize=3, writeQuorumSize=2, ackQuorumSize=2, state=CLOSED, length=161312417, lastEntryId=1492, digestType=CRC32C, password=base64:, ensembles={0=[112.90.117.244:3181, 112.90.117.245:3181, 112.90.117.22:3181]}, customMetadata={pulsar/managed-ledger=base64:cHVsc2FydGVzdDEzL3B1bHNhcnRlc3RuYW1lc3BhY2UxMy9wZXJzaXN0ZW50L3B1bHNhcnRlc3R0b3BpYzEzX21lbS1wYXJ0aXRpb24tMTEz, component=base64:bWFuYWdlZC1sZWRnZXI=, application=base64:cHVsc2Fy}}
00:43:42.665 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x30066ef5a1801c0 closed
00:43:42.665 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x30066ef5a1801c0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ericwangdka commented on issue #7214: bookie server's log : No ledger found while reading entry
Posted by GitBox <gi...@apache.org>.
ericwangdka commented on issue #7214:
URL: https://github.com/apache/pulsar/issues/7214#issuecomment-642144543
in fact ledger 1500467 cannot be found [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.client.PendingReadOp - Read of ledger entry failed: L1500467 E1338-E1437, Sent to [112.90.117.244:3181, 112.90.117.245:3181, 112.90.117.22:3181], Heard from [132.90.117.22:3181] : bitset = {2}, Error = 'No such ledger exists on Bookies'. First unread entry is (1339, rc = 0)
01:11:13.494 [bookkeeper-ml-workers-OrderedExecutor-2-0] WARN org.apache.bookkeeper.mledger.impl.OpReadEntry - [pulsartest13/pulsartestnamespace13/persistent/pulsartesttopic13_mem-partition-113][subscription-trace-mme] read failed from ledger at position:1500467:1338 : No such ledger exists on Bookies
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ericwangdka commented on issue #7214: bookie server's log : No ledger found while reading entry
Posted by GitBox <gi...@apache.org>.
ericwangdka commented on issue #7214:
URL: https://github.com/apache/pulsar/issues/7214#issuecomment-642149678
@sijie because of the network ,112.90.117.244:3181, 112.90.117.245:3181 restart at the same time ,how can i recovery it or skip it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Ghatage commented on issue #7214: bookie server's log : No ledger found while reading entry
Posted by GitBox <gi...@apache.org>.
Ghatage commented on issue #7214:
URL: https://github.com/apache/pulsar/issues/7214#issuecomment-642147438
@ericwangdka
> it appears ledger: 1500467 is corrupt,how can i delete it
If you absolutely want to delete the ledger, [the bookie shell provides facility to delete a ledger](https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/tools/cli/commands/client/DeleteLedgerCommand.java).
Not sure if I would recommend it in general as you must be sure about what you are doing then manually intervening.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ericwangdka commented on issue #7214: bookie server's log : No ledger found while reading entry
Posted by GitBox <gi...@apache.org>.
ericwangdka commented on issue #7214:
URL: https://github.com/apache/pulsar/issues/7214#issuecomment-641948403
@sijie however ，it appears ledger: 1500467 is corrupt,how can i delete it
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 312 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 315 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 318 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 321 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.client.PendingReadOp - Read of ledger entry failed: L1500467 E393-E393, Sent to [132.90.117.244:3181, 132.90.117.245:3181], Heard from [] : bitset = {}, Error = 'No such ledger exists on Bookies'. First unread entry is (-1, rc = null)
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 324 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.client.PendingReadOp - Read of ledger entry failed: L1500467 E396-E396, Sent to [132.90.117.244:3181, 132.90.117.245:3181], Heard from [] : bitset = {}, Error = 'No such ledger exists on Bookies'. First unread entry is (-1, rc = null)
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 327 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 330 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 333 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.client.PendingReadOp - Read of ledger entry failed: L1500467 E399-E399, Sent to [132.90.117.244:3181, 132.90.117.245:3181], Heard from [] : bitset = {}, Error = 'No such ledger exists on Bookies'. First unread entry is (-1, rc = null)
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 336 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 339 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 342 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 345 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 348 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 351 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 354 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 357 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 360 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 363 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 366 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 369 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 372 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 375 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 378 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 381 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 384 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 387 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 390 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 393 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 396 of ledger: 1500467 in ReplicationWorker
19:26:03.944 [BookKeeperClientWorker-OrderedExecutor-27-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 399 of ledger: 1500467 in ReplicationWorker
19:26:03.945 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 1500467 for 33 number of times, so deferring the ledger lock release by 300000 msecs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Ghatage edited a comment on issue #7214: bookie server's log : No ledger found while reading entry
Posted by GitBox <gi...@apache.org>.
Ghatage edited a comment on issue #7214:
URL: https://github.com/apache/pulsar/issues/7214#issuecomment-642147438
@ericwangdka
> it appears ledger: 1500467 is corrupt,how can i delete it
If you absolutely want to delete the ledger, [the bookie shell provides facility to delete a ledger](https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/tools/cli/commands/client/DeleteLedgerCommand.java).
Not sure if I would recommend it in general as you must be sure about what you are doing when manually intervening.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7214: bookie server's log : No ledger found while reading entry
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7214:
URL: https://github.com/apache/pulsar/issues/7214#issuecomment-641668136
@ericwangdka
The "No ledger found" is a INFO logging. (we can probably change that to reduce the logging level to avoid the noises).
"No ledger found" usually happens when brokers recover a managed ledger with an empty segment/ledger. That typically means your broker cluster is frequently changing the topic ownerships. Are you running any load testing to trigger broker load balancing the topics?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org