You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/09/15 17:25:12 UTC
[GitHub] [pulsar] evfurman opened a new issue #8069: Consumption Stops When Losing a Bookie.
evfurman opened a new issue #8069:
URL: https://github.com/apache/pulsar/issues/8069
**Describe the bug**
Consumers stop consuming when bookie service is stopped.
Slack thread: https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1600113955395000
We have a 4 node bookie cluster utilizing rack awareness, running version `2.6.1.` While testing failure scenarios we stopped the `pulsar-bookie` service on one of the nodes. In the Pulsar Manager consumption throughput dropped to zero across all partitioned topics. Production continued fine. This caused an immediate backlog (obviously). We restarted the `pulsar-bookie` service but consumption did not resume. @addisonj recommended trying to unload the partition which caused consumption to resume.
We are using the default namespace with the following config:
```
# Number of bookies to use when creating a ledger
managedLedgerDefaultEnsembleSize=3
# Number of copies to store for each message
managedLedgerDefaultWriteQuorum=2
# Number of guaranteed copies (acks to wait before write is complete)
managedLedgerDefaultAckQuorum=2
```
**Expected behavior**
We expected higher latencies briefly with the loss of a bookie but would have thought production/consumption would not stop working entirely.
**Screenshots**
Consumption stops.
<img width="1608" alt="Screen Shot 2020-09-14 at 4 31 57 PM" src="https://user-images.githubusercontent.com/7016955/93243653-1c264000-f74e-11ea-9ab9-bcf763b5da68.png">
After unloading partition, consumption resumes.
<img width="1550" alt="Screen Shot 2020-09-15 at 11 50 32 AM" src="https://user-images.githubusercontent.com/7016955/93243590-06b11600-f74e-11ea-834a-20f826ba62ad.png">
**Additional context**
Attaching logs from the brokers.
[pulsar-broker-1.log-09-14-2020-1.log.gz](https://github.com/apache/pulsar/files/5226610/pulsar-broker-1.log-09-14-2020-1.log.gz)
[pulsar-broker-2.log-09-14-2020-1.log.gz](https://github.com/apache/pulsar/files/5226611/pulsar-broker-2.log-09-14-2020-1.log.gz)
[pulsar-broker-3.log-09-14-2020-1.log.gz](https://github.com/apache/pulsar/files/5226612/pulsar-broker-3.log-09-14-2020-1.log.gz)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] evfurman commented on issue #8069: Consumption Stops When Losing a Bookie.
Posted by GitBox <gi...@apache.org>.
evfurman commented on issue #8069:
URL: https://github.com/apache/pulsar/issues/8069#issuecomment-693687785
`Key_Shared`
We're using the pulsar-perf with the following arguments:
`/pulsar/bin/pulsar-perf consume persistent://public/default/mp-explicit-partitioned --subscriber-name mp-pulsar-consumer -t 8 -st Key_Shared -u pulsar://pulsar.qa.corp.mparticle.com:6650`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #8069: Consumption Stops When Losing a Bookie.
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #8069:
URL: https://github.com/apache/pulsar/issues/8069#issuecomment-693775694
@evfurman If you are using Key_Shared, you might be impacted by https://github.com/apache/pulsar/pull/8024
You can try the hotfix:
https://github.com/streamnative/pulsar/releases/tag/v2.6.1-sn-3
See discussions on https://github.com/apache/pulsar/issues/7682#issuecomment-693251109
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] evfurman edited a comment on issue #8069: Consumption Stops When Losing a Bookie.
Posted by GitBox <gi...@apache.org>.
evfurman edited a comment on issue #8069:
URL: https://github.com/apache/pulsar/issues/8069#issuecomment-693687785
`Key_Shared`
We're using the pulsar-perf with the following arguments:
`/pulsar/bin/pulsar-perf consume persistent://public/default/mp-explicit-partitioned --subscriber-name mp-pulsar-consumer -t 8 -st Key_Shared -u pulsar://pulsar.xx.xx.xx.com:6650`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #8069: Consumption Stops When Losing a Bookie.
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #8069:
URL: https://github.com/apache/pulsar/issues/8069#issuecomment-693122291
@jiazhai Can you check the broker logs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #8069: Consumption Stops When Losing a Bookie.
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #8069:
URL: https://github.com/apache/pulsar/issues/8069#issuecomment-693122415
@evfurman what type of subscription are you using?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org