You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/07/22 16:57:14 UTC
[GitHub] [pulsar] frankjkelly opened a new issue #11434: Pulsar Upgrade from 2.6.1 to 2.7.2: Bookie reporting "No ledger found while performing readLac from ledger"
frankjkelly opened a new issue #11434:
URL: https://github.com/apache/pulsar/issues/11434
**Describe the bug**
We performed an in-place upgrade of Pulsar 2.6.1 to Pulsar 2.7.2. Although it appears to be functional overall I am seeing the following in the Bookie logs
{noformat}
platform-pulsar-bookkeeper-16 platform-pulsar-bookkeeper 16:53:08.577 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.proto.ReadLacProcessorV3 - No ledger found while performing readLac from ledger: 2392970
platform-pulsar-bookkeeper-16 platform-pulsar-bookkeeper org.apache.bookkeeper.bookie.Bookie$NoLedgerException: Ledger 2392970 not found
platform-pulsar-bookkeeper-16 platform-pulsar-bookkeeper 16:53:08.577 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.proto.ReadLacProcessorV3 - No ledger found while trying to read last entry: 2392970
platform-pulsar-bookkeeper-16 platform-pulsar-bookkeeper org.apache.bookkeeper.bookie.Bookie$NoLedgerException: Ledger 2392970 not found
platform-pulsar-bookkeeper-11 platform-pulsar-bookkeeper 16:53:08.577 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.proto.ReadLacProcessorV3 - No ledger found while performing readLac from ledger: 2392970
platform-pulsar-bookkeeper-11 platform-pulsar-bookkeeper org.apache.bookkeeper.bookie.Bookie$NoLedgerException: Ledger 2392970 not found
platform-pulsar-bookkeeper-11 platform-pulsar-bookkeeper 16:53:08.577 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.proto.ReadLacProcessorV3 - No ledger found while trying to read last entry: 2392970
platform-pulsar-bookkeeper-11 platform-pulsar-bookkeeper org.apache.bookkeeper.bookie.Bookie$NoLedgerException: Ledger 2392970 not found
platform-pulsar-bookkeeper-11 platform-pulsar-bookkeeper 16:53:08.578 [BookieReadThreadPool-OrderedExecutor-2-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 0 from ledger: 2392970
platform-pulsar-bookkeeper-11 platform-pulsar-bookkeeper 16:53:08.579 [BookieReadThreadPool-OrderedExecutor-2-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 0 from ledger: 2392970
platform-pulsar-bookkeeper-13 platform-pulsar-bookkeeper 16:53:08.577 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.proto.ReadLacProcessorV3 - No ledger found while performing readLac from ledger: 2392970
platform-pulsar-bookkeeper-13 platform-pulsar-bookkeeper org.apache.bookkeeper.bookie.Bookie$NoLedgerException: Ledger 2392970 not found
platform-pulsar-bookkeeper-13 platform-pulsar-bookkeeper 16:53:08.577 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.proto.ReadLacProcessorV3 - No ledger found while trying to read last entry: 2392970
platform-pulsar-bookkeeper-13 platform-pulsar-bookkeeper org.apache.bookkeeper.bookie.Bookie$NoLedgerException: Ledger 2392970 not found
platform-pulsar-bookkeeper-9 platform-pulsar-bookkeeper 16:53:08.577 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.proto.ReadLacProcessorV3 - No ledger found while performing readLac from ledger: 2392970
platform-pulsar-bookkeeper-9 platform-pulsar-bookkeeper org.apache.bookkeeper.bookie.Bookie$NoLedgerException: Ledger 2392970 not found
platform-pulsar-bookkeeper-9 platform-pulsar-bookkeeper 16:53:08.577 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.proto.ReadLacProcessorV3 - No ledger found while trying to read last entry: 2392970
platform-pulsar-bookkeeper-9 platform-pulsar-bookkeeper org.apache.bookkeeper.bookie.Bookie$NoLedgerException: Ledger 2392970 not found
platform-pulsar-bookkeeper-14 platform-pulsar-bookkeeper 16:53:08.577 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.proto.ReadLacProcessorV3 - No ledger found while performing readLac from ledger: 2392970
platform-pulsar-bookkeeper-14 platform-pulsar-bookkeeper org.apache.bookkeeper.bookie.Bookie$NoLedgerException: Ledger 2392970 not found
platform-pulsar-bookkeeper-14 platform-pulsar-bookkeeper 16:53:08.577 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.proto.ReadLacProcessorV3 - No ledger found while trying to read last entry: 2392970
platform-pulsar-bookkeeper-14 platform-pulsar-bookkeeper org.apache.bookkeeper.bookie.Bookie$NoLedgerException: Ledger 2392970 not found
platform-pulsar-bookkeeper-0 platform-pulsar-bookkeeper 16:53:08.577 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.proto.ReadLacProcessorV3 - No ledger found while performing readLac from ledger: 2392970
platform-pulsar-bookkeeper-0 platform-pulsar-bookkeeper org.apache.bookkeeper.bookie.Bookie$NoLedgerException: Ledger 2392970 not found
platform-pulsar-bookkeeper-0 platform-pulsar-bookkeeper 16:53:08.577 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.proto.ReadLacProcessorV3 - No ledger found while trying to read last entry: 2392970
platform-pulsar-bookkeeper-0 platform-pulsar-bookkeeper org.apache.bookkeeper.bookie.Bookie$NoLedgerException: Ledger 2392970 not found
platform-pulsar-bookkeeper-10 platform-pulsar-bookkeeper 16:53:08.577 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.proto.ReadLacProcessorV3 - No ledger found while performing readLac from ledger: 2392970
platform-pulsar-bookkeeper-10 platform-pulsar-bookkeeper org.apache.bookkeeper.bookie.Bookie$NoLedgerException: Ledger 2392970 not found
platform-pulsar-bookkeeper-10 platform-pulsar-bookkeeper 16:53:08.577 [BookieReadThreadPool-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.proto.ReadLacProcessorV3 - No ledger found while trying to read last entry: 2392970
platform-pulsar-bookkeeper-10 platform-pulsar-bookkeeper org.apache.bookkeeper.bookie.Bookie$NoLedgerException: Ledger 2392970 not found
platform-pulsar-bookkeeper-10 platform-pulsar-bookkeeper 16:53:08.578 [BookieReadThreadPool-OrderedExecutor-2-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 0 from ledger: 2392970
platform-pulsar-bookkeeper-10 platform-pulsar-bookkeeper 16:53:08.579 [BookieReadThreadPool-OrderedExecutor-2-0] INFO org.apache.bookkeeper.proto.ReadEntryProcessorV3 - No ledger found while reading entry: 0 from ledger: 2392970
{noformat}
**To Reproduce**
Steps to reproduce the behavior:
1. Create a Pulsar 2.6.2 Cluster
2. Setup Topics and write data to it
3. Upgrade in-place to 2.7.2 Pulsar
**Expected behavior**
No repeated errors in Bookie log files
**Screenshots**
See logs above
**Desktop (please complete the following information):**
- OS: Kubernetes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly closed issue #11434: Pulsar Upgrade from 2.6.1 to 2.7.2: Bookie reporting "No ledger found while performing readLac from ledger"
Posted by GitBox <gi...@apache.org>.
frankjkelly closed issue #11434:
URL: https://github.com/apache/pulsar/issues/11434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly commented on issue #11434: Pulsar Upgrade from 2.6.1 to 2.7.2: Bookie reporting "No ledger found while performing readLac from ledger"
Posted by GitBox <gi...@apache.org>.
frankjkelly commented on issue #11434:
URL: https://github.com/apache/pulsar/issues/11434#issuecomment-1049205529
Closing - I think this was because we had an ACK_QUORUM of 1. Either way it's not reproducible any more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly commented on issue #11434: Pulsar Upgrade from 2.6.1 to 2.7.2: Bookie reporting "No ledger found while performing readLac from ledger"
Posted by GitBox <gi...@apache.org>.
frankjkelly commented on issue #11434:
URL: https://github.com/apache/pulsar/issues/11434#issuecomment-887813356
Unfortunately no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly edited a comment on issue #11434: Pulsar Upgrade from 2.6.1 to 2.7.2: Bookie reporting "No ledger found while performing readLac from ledger"
Posted by GitBox <gi...@apache.org>.
frankjkelly edited a comment on issue #11434:
URL: https://github.com/apache/pulsar/issues/11434#issuecomment-887813356
Unfortunately no
UPDATE: Just to clarify it's data from analysis of real-time audio and those audio signals are now "gone" (they themselves were on a topic but it was not retained).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly commented on issue #11434: Pulsar Upgrade from 2.6.1 to 2.7.2: Bookie reporting "No ledger found while performing readLac from ledger"
Posted by GitBox <gi...@apache.org>.
frankjkelly commented on issue #11434:
URL: https://github.com/apache/pulsar/issues/11434#issuecomment-887779809
Thanks @eolivelli
I used the following command to get the metadata about the ledgerId being complained about
```
./bookkeeper shell ledgermetadata -ledgerid 2658812
```
and I get
```
.
.
.
19:27:28.026 [main] INFO org.apache.bookkeeper.tools.cli.commands.client.LedgerMetaDataCommand - ledgerID: 2658812
19:27:28.031 [main] INFO org.apache.bookkeeper.tools.cli.commands.client.LedgerMetaDataCommand - LedgerMetadata{formatVersion=3, ensembleSize=7, writeQuorumSize=2, ackQuorumSize=1, state=OPEN, digestType=CRC32C, password=base64:, ensembles={0=[platform-pulsar-bookkeeper-17.platform-pulsar-bookkeeper.cogito-saas.svc.cluster.local:3181, platform-pulsar-bookkeeper-4.platform-pulsar-bookkeeper.cogito-saas.svc.cluster.local:3181, platform-pulsar-bookkeeper-16.platform-pulsar-bookkeeper.cogito-saas.svc.cluster.local:3181, platform-pulsar-bookkeeper-6.platform-pulsar-bookkeeper.cogito-saas.svc.cluster.local:3181, platform-pulsar-bookkeeper-2.platform-pulsar-bookkeeper.cogito-saas.svc.cluster.local:3181, platform-pulsar-bookkeeper-3.platform-pulsar-bookkeeper.cogito-saas.svc.cluster.local:3181, platform-pulsar-bookkeeper-8.platform-pulsar-bookkeeper.cogito-saas.svc.cluster.local:3181]}, customMetadata={component=base64:bWFuYWdlZC1sZWRnZXI=, pulsar/managed-ledger=base64:Y29naXRvLWRpYWxvZy
9zdGYvcGVyc2lzdGVudC9lYzVmMWQxMC1lNDAyLTQyMDEtYmUwOS1jMWE0NmUzZDJiM2Y=, application=base64:cHVsc2Fy}}
19:27:28.135 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x204a5547020008d closed
```
If I Base64 decode the metadata I get
- component: managed-ledger
- ledger: cogito-dialog/stf/persistent/ec5f1d10-e402-4201-be09-c1a46e3d2b3f
- application: pulsar
So it looks to be one of our analytics data topics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #11434: Pulsar Upgrade from 2.6.1 to 2.7.2: Bookie reporting "No ledger found while performing readLac from ledger"
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #11434:
URL: https://github.com/apache/pulsar/issues/11434#issuecomment-887800354
Is it possible to drop the topic and recreate it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #11434: Pulsar Upgrade from 2.6.1 to 2.7.2: Bookie reporting "No ledger found while performing readLac from ledger"
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #11434:
URL: https://github.com/apache/pulsar/issues/11434#issuecomment-887461461
do you have a way to see what it ledger 2392970 ?
usually ledger metadata on zookeeper should tell you what is the purpose of the ledger
you can use the bookkeeper "shell" or a tool like BKVM (https://github.com/diennea/bookkeeper-visual-manager)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org