You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/04/06 14:04:34 UTC
[GitHub] [pulsar] archmangler opened a new issue, #15045: reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
archmangler opened a new issue, #15045:
URL: https://github.com/apache/pulsar/issues/15045
**Describe the bug**
Connecting to pulsar with a Golang code example results in the following error:
```
reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
```
Complete context:
```
root@pulsar-toolset-0:/pulsar/development# ./pulse
2022/04/06 13:58:50.688 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/06 13:58:50.710 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:45880 -> 192.168.3.142:6650] Connected to broker
2022/04/06 13:58:50.716 c_client.go:68: [info] INFO | HandlerBase:54 | [persistent://ragnarok/transactions/requests, reader-5ada8746c3, 0] Getting connection from pool
2022/04/06 13:58:50.718 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker-1.pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/06 13:58:50.720 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:36066 -> 192.168.6.237:6650] Connected to broker
2022/04/06 13:58:50.723 c_client.go:68: [info] WARN | ClientConnection:971 | [192.168.4.54:36066 -> 192.168.6.237:6650] Received error response from server: BrokerPersistenceError -- req_id: 0
2022/04/06 13:58:50.723 c_client.go:68: [info] ERROR | ConsumerImpl:263 | [persistent://ragnarok/transactions/requests, reader-5ada8746c3, 0] Failed to create consumer: BrokerPersistenceError
2022/04/06 13:58:50 Could not create reader: Failed to create Reader: BrokerPersistenceError
```
I have been getting this error consistently when using the example Golang code for a Reader (here: https://pulsar.apache.org/docs/en/2.3.1/client-libraries-go/#readers).
To recover, I need to restart the brokers with something like:
```
kubectl rollout restart sts pulsar-broker -n pulsar
```
**To Reproduce**
Steps to reproduce the behavior:
1. Copy the `Reader` code from https://pulsar.apache.org/docs/en/2.3.1/client-libraries-go/#readers, go build <scriptname>:
```
import (
"context"
"log"
"github.com/apache/pulsar/pulsar-client-go/pulsar"
)
func main() {
// Instantiate a Pulsar client
client, err := pulsar.NewClient(pulsar.ClientOptions{
URL: "pulsar://localhost:6650",
})
if err != nil { log.Fatalf("Could not create client: %v", err) }
// Use the client to instantiate a reader
reader, err := client.CreateReader(pulsar.ReaderOptions{
Topic: "my-golang-topic",
StartMessageID: pulsar.EarliestMessage,
})
if err != nil { log.Fatalf("Could not create reader: %v", err) }
defer reader.Close()
ctx := context.Background()
// Listen on the topic for incoming messages
for {
msg, err := reader.Next(ctx)
if err != nil { log.Fatalf("Error reading from topic: %v", err) }
// Process the message
}
}
```
3. Run the Golang client reader once to connect to pulsar.
4. Cancel the client (ctrl-c)
5. Try rerun the client again
**Expected behavior**
The client should cancel gracefully and when rerun connect to pulsar broker successfully.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: Alpine Docker container on Kubernetes.
**Additional context**
Add any other context about the problem here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tjiuming commented on issue #15045: reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
Posted by GitBox <gi...@apache.org>.
tjiuming commented on issue #15045:
URL: https://github.com/apache/pulsar/issues/15045#issuecomment-1090443210
```
func main() {
//end of read stream
stop := 16
// Instantiate a Pulsar client
client, err := pulsar.NewClient(pulsar.ClientOptions{
URL: "pulsar://pulsar-broker.pulsar.svc.cluster.local:6650",
})
if err != nil {
log.Fatalf("Could not create client: %v", err)
}
// Use the client to instantiate a reader
reader, err := client.CreateReader(pulsar.ReaderOptions{
Topic: "ragnarok/transactions/requests",
StartMessageID: pulsar.EarliestMessage,
})
if err != nil {
log.Fatalf("Could not create reader: %v", err)
}
ctx := context.Background()
// Listen on the topic for incoming messages
cnt := 0
for {
msg, err := reader.Next(ctx)
if err != nil {
log.Fatalf("Error reading from topic: %v", err)
} else {
cnt++
content := string(msg.Payload())
fmt.Println(cnt, " -> ", msg.ID(), " -> ", content)
if cnt == stop {
fmt.Println("reached stream end. breaking out.")
defer reader.Close()
defer client.Close()
break
}
}
}
}
```
like this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tjiuming commented on issue #15045: reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
Posted by GitBox <gi...@apache.org>.
tjiuming commented on issue #15045:
URL: https://github.com/apache/pulsar/issues/15045#issuecomment-1090330079
Actually, it depends. Most of this error should be happened in `ledger ops` or `cursor ops`. Broker side should has error/warning log and you can check it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tjiuming commented on issue #15045: reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
Posted by GitBox <gi...@apache.org>.
tjiuming commented on issue #15045:
URL: https://github.com/apache/pulsar/issues/15045#issuecomment-1090431022
You can close client in defer scope and retry. Seems it's netty's feature. In some situation, Channel#inactive will be invoked more than once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on issue #15045: reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on issue #15045:
URL: https://github.com/apache/pulsar/issues/15045#issuecomment-1090357605
The stack trace may be optimized by JVM. Could you please share more broker logs around this time ? or you can grep broker logs to find NPE error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] archmangler commented on issue #15045: reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
Posted by GitBox <gi...@apache.org>.
archmangler commented on issue #15045:
URL: https://github.com/apache/pulsar/issues/15045#issuecomment-1090337911
Thanks @tjiuming for your response. On the broker side I see this:
```
pulsar-broker-0 pulsar-broker 2022-04-06T22:26:05.869117680+08:00 14:26:05.868 [pulsar-io-24-2] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /192.168.4.54:36310
pulsar-broker-2 pulsar-broker 2022-04-06T22:26:05.876304494+08:00 14:26:05.876 [pulsar-io-24-1] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /192.168.4.54:48210
pulsar-broker-2 pulsar-broker 2022-04-06T22:26:05.876877877+08:00 14:26:05.876 [pulsar-io-24-1] INFO org.apache.pulsar.broker.service.ServerCnx - [/192.168.4.54:48210] Subscribing on topic persistent://ragnarok/transactions/requests / reader-d6ddeb848f
pulsar-broker-2 pulsar-broker 2022-04-06T22:26:05.877030498+08:00 14:26:05.876 [pulsar-io-24-1] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://ragnarok/transactions/requests][reader-d6ddeb848f] Creating non-durable subscription at msg id -1:-1:-1:-1
pulsar-broker-2 pulsar-broker 2022-04-06T22:26:05.877233753+08:00 14:26:05.877 [pulsar-io-24-1] INFO org.apache.bookkeeper.mledger.impl.NonDurableCursorImpl - [ragnarok/transactions/persistent/requests] Created non-durable cursor read-position=162:0 mark-delete-position=162:-1
pulsar-broker-2 pulsar-broker 2022-04-06T22:26:05.877311049+08:00 14:26:05.877 [pulsar-io-24-1] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [ragnarok/transactions/persistent/requests] Opened new cursor: NonDurableCursorImpl{ledger=ragnarok/transactions/persistent/requests, ackPos=162:-1, readPos=162:0}
pulsar-broker-2 pulsar-broker 2022-04-06T22:26:05.877443482+08:00 14:26:05.877 [pulsar-io-24-1] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ragnarok/transactions/persistent/requests-reader-d6ddeb848f] Rewind from 162:0 to 162:0
pulsar-broker-2 pulsar-broker 2022-04-06T22:26:05.877834039+08:00 14:26:05.877 [pulsar-io-24-1] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://ragnarok/transactions/requests] Failed to create subscription: reader-d6ddeb848f error: java.util.concurrent.CompletionException: java.lang.NullPointerException
pulsar-broker-2 pulsar-broker 2022-04-06T22:26:05.877848619+08:00 14:26:05.877 [pulsar-io-24-1] WARN org.apache.pulsar.broker.service.ServerCnx - [/192.168.4.54:48210][persistent://ragnarok/transactions/requests][reader-d6ddeb848f] Failed to create consumer: consumerId=0, java.util.concurrent.CompletionException: java.lang.NullPointerException
pulsar-broker-2 pulsar-broker 2022-04-06T22:26:05.878862443+08:00 14:26:05.878 [pulsar-io-24-1] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /192.168.4.54:48210
pulsar-broker-0 pulsar-broker 2022-04-06T22:26:05.879334129+08:00 14:26:05.879 [pulsar-io-24-2] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /192.168.4.54:36310
pulsar-broker-0 pulsar-broker 2022-04-06T22:26:07.338668498+08:00 14:26:07.338 [pulsar-load-manager-2-1] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Writing local data to ZooKeeper because maximum change 11.808785796165466% exceeded threshold 10%; time since last report written is 180.0 seconds
pulsar-broker-0 pulsar-broker 2022-04-06T22:26:07.348366859+08:00 14:26:07.348 [pulsar-ordered-OrderedExecutor-6-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x200003ccebc0097 local:/192.168.2.72:52102 remoteserver:pulsar-zookeeper/192.168.5.171:2181 lastZxid:4294985310 xid:777 sent:777 recv:843 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: WatchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080
pulsar-broker-2 pulsar-broker 2022-04-06T22:26:07.349049969+08:00 14:26:07.348 [pulsar-ordered-OrderedExecutor-6-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x100003ceea30079 local:/192.168.5.36:33404 remoteserver:pulsar-zookeeper/192.168.2.172:2181 lastZxid:4294985309 xid:1077 sent:1077 recv:1163 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: WatchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080
pulsar-broker-1 pulsar-broker 2022-04-06T22:26:07.348748508+08:00 14:26:07.348 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x300003cf05d007d local:/192.168.6.225:50406 remoteserver:pulsar-zookeeper/192.168.7.44:2181 lastZxid:4294985309 xid:822 sent:822 recv:900 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: WatchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080
pulsar-broker-2 pulsar-broker 2022-04-06T22:26:09.340012935+08:00 14:26:09.336 [pulsar-web-42-6] INFO org.eclipse.jetty.server.RequestLog - 192.168.4.44 - - [06/Apr/2022:14:26:09 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/1.21+" 1
pulsar-broker-2 pulsar-broker 2022-04-06T22:26:09.340058219+08:00 14:26:09.336 [pulsar-web-42-8] INFO org.eclipse.jetty.server.RequestLog - 192.168.4.44 - - [06/Apr/2022:14:26:09 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/1.21+" 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- closed issue #15045: reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
Posted by GitBox <gi...@apache.org>.
Technoboy- closed issue #15045: reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
URL: https://github.com/apache/pulsar/issues/15045
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on issue #15045: reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on issue #15045:
URL: https://github.com/apache/pulsar/issues/15045#issuecomment-1090386525
This issue has been fixed by https://github.com/apache/pulsar/pull/12297. You can cherry pick to your local branch or upgrade to 2.7.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tjiuming commented on issue #15045: reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
Posted by GitBox <gi...@apache.org>.
tjiuming commented on issue #15045:
URL: https://github.com/apache/pulsar/issues/15045#issuecomment-1091041428
I read broker side source code yesterday, it's just a simple issue, upgrade broker to higher version can fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tjiuming commented on issue #15045: reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
Posted by GitBox <gi...@apache.org>.
tjiuming commented on issue #15045:
URL: https://github.com/apache/pulsar/issues/15045#issuecomment-1090351044
No stack trace. Can you find stacktrace in `pulsar/logs` directory?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] archmangler commented on issue #15045: reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
Posted by GitBox <gi...@apache.org>.
archmangler commented on issue #15045:
URL: https://github.com/apache/pulsar/issues/15045#issuecomment-1091256379
I've run the upgrade using the helm chart (it's at revision 3 now):
```
(base) welcome@Traianos-MacBook-Pro apache-pulsar % ./update.sh
W0407 15:24:25.809512 30274 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
Release "pulsar" has been upgraded. Happy Helming!
NAME: pulsar
LAST DEPLOYED: Thu Apr 7 15:24:24 2022
NAMESPACE: pulsar
STATUS: deployed
REVISION: 3
TEST SUITE: None
```
However, with a fresh run:
```
root@pulsar-toolset-0:/pulsar/development# ./pulse
2022/04/07 08:03:21.761 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/07 08:03:21.789 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:39724 -> 192.168.2.72:6650] Connected to broker
2022/04/07 08:03:21.800 c_client.go:68: [info] INFO | HandlerBase:54 | [persistent://ragnarok/transactions/requests, reader-cb90910902, 0] Getting connection from pool
2022/04/07 08:03:21.805 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker-1.pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/07 08:03:21.807 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:37658 -> 192.168.6.78:6650] Connected to broker
2022/04/07 08:03:21.819 c_client.go:68: [info] INFO | ConsumerImpl:220 | [persistent://ragnarok/transactions/requests, reader-cb90910902, 0] Created consumer on broker [192.168.4.54:37658 -> 192.168.6.78:6650]
1 -> (162,0,-1,0) -> [redacted]
2 ->
.
.
.
16 -> (162,15,-1,0) -> [redacted]
reached stream end. breaking out.
2022/04/07 08:03:21.973 c_client.go:68: [info] INFO | ClientImpl:483 | Closing Pulsar client
2022/04/07 08:03:21.973 c_client.go:68: [info] INFO | ConsumerImpl:871 | [persistent://ragnarok/transactions/requests, reader-cb90910902, 0] Closing consumer for topic persistent://ragnarok/transactions/requests
2022/04/07 08:03:21.994 c_client.go:68: [info] ERROR | ClientConnection:523 | [192.168.4.54:37658 -> 192.168.6.78:6650] Read failed: End of file
2022/04/07 08:03:21.995 c_client.go:68: [info] INFO | ClientConnection:1436 | [192.168.4.54:37658 -> 192.168.6.78:6650] Connection closed
2022/04/07 08:03:21.995 c_client.go:68: [info] ERROR | ConsumerImpl:929 | [persistent://ragnarok/transactions/requests, reader-cb90910902, 0] Failed to close consumer: ConnectError
2022/04/07 08:03:21.995 c_client.go:68: [info] INFO | ClientConnection:1436 | [192.168.4.54:39724 -> 192.168.2.72:6650] Connection closed
```
Second run:
```
root@pulsar-toolset-0:/pulsar/development# ./pulse
2022/04/07 08:06:45.637 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/07 08:06:45.651 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:57400 -> 192.168.4.116:6650] Connected to broker
2022/04/07 08:06:45.656 c_client.go:68: [info] INFO | HandlerBase:54 | [persistent://ragnarok/transactions/requests, reader-23527b3907, 0] Getting connection from pool
2022/04/07 08:06:45.658 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker-1.pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/07 08:06:45.660 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:38658 -> 192.168.6.78:6650] Connected to broker
2022/04/07 08:06:45.669 c_client.go:68: [info] WARN | ClientConnection:971 | [192.168.4.54:38658 -> 192.168.6.78:6650] Received error response from server: BrokerPersistenceError -- req_id: 0
2022/04/07 08:06:45.669 c_client.go:68: [info] ERROR | ConsumerImpl:263 | [persistent://ragnarok/transactions/requests, reader-23527b3907, 0] Failed to create consumer: BrokerPersistenceError
2022/04/07 08:06:45 Could not create reader: Failed to create Reader: BrokerPersistenceError
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] archmangler commented on issue #15045: reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
Posted by GitBox <gi...@apache.org>.
archmangler commented on issue #15045:
URL: https://github.com/apache/pulsar/issues/15045#issuecomment-1090369186
Hi @tjiuming - attached is a broker log, after 1 run of the reader script, followed by a ctrl-c, then 3 attempts to connect using the reader script again: https://rentry.co/zpa29
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] archmangler commented on issue #15045: reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
Posted by GitBox <gi...@apache.org>.
archmangler commented on issue #15045:
URL: https://github.com/apache/pulsar/issues/15045#issuecomment-1090437213
Hi @tjiuming
`You can close client in defer scope and retry. `
Would you perhaps have a golang example for how to do this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] archmangler commented on issue #15045: reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
Posted by GitBox <gi...@apache.org>.
archmangler commented on issue #15045:
URL: https://github.com/apache/pulsar/issues/15045#issuecomment-1090407287
@tjiuming I can reproduce the issue at will by ctrl-c of the client after a read, it's as if the client does not terminate properly , leaving the broker in an inconsistent state:
See demonstration below:
- Run script:
```
root@pulsar-toolset-0:/pulsar/development# ./pulse
2022/04/06 15:23:41.601 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/06 15:23:41.607 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:59504 -> 192.168.6.203:6650] Connected to broker
2022/04/06 15:23:41.612 c_client.go:68: [info] INFO | HandlerBase:54 | [persistent://ragnarok/transactions/requests, reader-ac05b14cf9, 0] Getting connection from pool
2022/04/06 15:23:41.616 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker-2.pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/06 15:23:41.618 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:50092 -> 192.168.5.242:6650] Connected to broker
2022/04/06 15:23:41.628 c_client.go:68: [info] INFO | ConsumerImpl:220 | [persistent://ragnarok/transactions/requests, reader-ac05b14cf9, 0] Created consumer on broker [192.168.4.54:50092 -> 192.168.5.242:6650]
1 -> (162,0,-1,0) -> [{redacted"}]
2 ->
.
.
.
16 -> (162,15,-1,0) -> [{redacted]
reached stream end. breaking out.
```
- Rerun script:
```
root@pulsar-toolset-0:/pulsar/development# ./pulse
2022/04/06 15:25:31.951 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/06 15:25:31.969 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:50590 -> 192.168.5.242:6650] Connected to broker
2022/04/06 15:25:31.973 c_client.go:68: [info] INFO | HandlerBase:54 | [persistent://ragnarok/transactions/requests, reader-9316fe9b08, 0] Getting connection from pool
2022/04/06 15:25:31.974 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker-2.pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/06 15:25:31.975 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:50592 -> 192.168.5.242:6650] Connected to broker
2022/04/06 15:25:31.979 c_client.go:68: [info] WARN | ClientConnection:971 | [192.168.4.54:50592 -> 192.168.5.242:6650] Received error response from server: BrokerPersistenceError -- req_id: 0
2022/04/06 15:25:31.979 c_client.go:68: [info] ERROR | ConsumerImpl:263 | [persistent://ragnarok/transactions/requests, reader-9316fe9b08, 0] Failed to create consumer: BrokerPersistenceError
2022/04/06 15:25:31 Could not create reader: Failed to create Reader: BrokerPersistenceError
```
- Restart brokers. Then run again (NOTE: I always see `Failed to close consumer: ConnectError`) : on the first rurn
```
root@pulsar-toolset-0:/pulsar/development# ./pulse
2022/04/06 15:27:07.893 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/06 15:27:07.899 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:45236 -> 192.168.4.194:6650] Connected to broker
2022/04/06 15:27:07.904 c_client.go:68: [info] INFO | HandlerBase:54 | [persistent://ragnarok/transactions/requests, reader-6b2a574e90, 0] Getting connection from pool
2022/04/06 15:27:07.908 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/06 15:27:07.910 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:46276 -> 192.168.2.18:6650] Connected to broker
2022/04/06 15:27:07.923 c_client.go:68: [info] INFO | ConsumerImpl:220 | [persistent://ragnarok/transactions/requests, reader-6b2a574e90, 0] Created consumer on broker [192.168.4.54:46276 -> 192.168.2.18:6650]
1 -> (162,0,-1,0) -> [{redacted}]
2 -> (162,1,-1,0) ->
.
.
16 -> (162,15,-1,0) -> [{redacted}]
reached stream end. breaking out.
2022/04/06 15:27:08.099 c_client.go:68: [info] INFO | ConsumerImpl:871 | [persistent://ragnarok/transactions/requests, reader-6b2a574e90, 0] Closing consumer for topic persistent://ragnarok/transactions/requests
2022/04/06 15:27:08.131 c_client.go:68: [info] ERROR | ClientConnection:523 | [192.168.4.54:46276 -> 192.168.2.18:6650] Read failed: End of file
2022/04/06 15:27:08.131 c_client.go:68: [info] INFO | ClientConnection:1436 | [192.168.4.54:46276 -> 192.168.2.18:6650] Connection closed
2022/04/06 15:27:08.131 c_client.go:68: [info] ERROR | ConsumerImpl:929 | [persistent://ragnarok/transactions/requests, reader-6b2a574e90, 0] Failed to close consumer: ConnectError
root@pulsar-toolset-0:/pulsar/development#
root@pulsar-toolset-0:/pulsar/development#
```
Run multiples more:
```
root@pulsar-toolset-0:/pulsar/development# ./pulse
2022/04/06 15:29:09.238 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/06 15:29:09.243 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:45784 -> 192.168.4.194:6650] Connected to broker
2022/04/06 15:29:09.247 c_client.go:68: [info] INFO | HandlerBase:54 | [persistent://ragnarok/transactions/requests, reader-1c1ee3b179, 0] Getting connection from pool
2022/04/06 15:29:09.248 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker-2.pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/06 15:29:09.249 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:45786 -> 192.168.4.194:6650] Connected to broker
2022/04/06 15:29:09.256 c_client.go:68: [info] WARN | ClientConnection:971 | [192.168.4.54:45786 -> 192.168.4.194:6650] Received error response from server: BrokerPersistenceError -- req_id: 0
2022/04/06 15:29:09.256 c_client.go:68: [info] ERROR | ConsumerImpl:263 | [persistent://ragnarok/transactions/requests, reader-1c1ee3b179, 0] Failed to create consumer: BrokerPersistenceError
2022/04/06 15:29:09 Could not create reader: Failed to create Reader: BrokerPersistenceError
root@pulsar-toolset-0:/pulsar/development#
root@pulsar-toolset-0:/pulsar/development#
root@pulsar-toolset-0:/pulsar/development# ./pulse
2022/04/06 15:29:10.500 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/06 15:29:10.516 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:46718 -> 192.168.2.70:6650] Connected to broker
2022/04/06 15:29:10.526 c_client.go:68: [info] INFO | HandlerBase:54 | [persistent://ragnarok/transactions/requests, reader-925c2cbe73, 0] Getting connection from pool
2022/04/06 15:29:10.530 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker-2.pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/06 15:29:10.532 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:45790 -> 192.168.4.194:6650] Connected to broker
2022/04/06 15:29:10.535 c_client.go:68: [info] WARN | ClientConnection:971 | [192.168.4.54:45790 -> 192.168.4.194:6650] Received error response from server: BrokerPersistenceError -- req_id: 0
2022/04/06 15:29:10.535 c_client.go:68: [info] ERROR | ConsumerImpl:263 | [persistent://ragnarok/transactions/requests, reader-925c2cbe73, 0] Failed to create consumer: BrokerPersistenceError
2022/04/06 15:29:10 Could not create reader: Failed to create Reader: BrokerPersistenceError
root@pulsar-toolset-0:/pulsar/development#
root@pulsar-toolset-0:/pulsar/development#
root@pulsar-toolset-0:/pulsar/development# ./pulse
2022/04/06 15:29:11.987 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/06 15:29:11.993 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:35980 -> 192.168.6.87:6650] Connected to broker
2022/04/06 15:29:11.999 c_client.go:68: [info] INFO | HandlerBase:54 | [persistent://ragnarok/transactions/requests, reader-6d58cde4d9, 0] Getting connection from pool
2022/04/06 15:29:12.000 c_client.go:68: [info] INFO | ConnectionPool:84 | Created connection for pulsar://pulsar-broker-2.pulsar-broker.pulsar.svc.cluster.local:6650
2022/04/06 15:29:12.002 c_client.go:68: [info] INFO | ClientConnection:364 | [192.168.4.54:45794 -> 192.168.4.194:6650] Connected to broker
2022/04/06 15:29:12.005 c_client.go:68: [info] WARN | ClientConnection:971 | [192.168.4.54:45794 -> 192.168.4.194:6650] Received error response from server: BrokerPersistenceError -- req_id: 0
2022/04/06 15:29:12.005 c_client.go:68: [info] ERROR | ConsumerImpl:263 | [persistent://ragnarok/transactions/requests, reader-6d58cde4d9, 0] Failed to create consumer: BrokerPersistenceError
2022/04/06 15:29:12 Could not create reader: Failed to create Reader: BrokerPersistenceError
root@pulsar-toolset-0:/pulsar/development#
root@pulsar-toolset-0:/pulsar/development#
```
Restarting the brokers allows us to connect once again.
The test code:
```
package main
import (
"context"
"fmt"
"log"
"github.com/apache/pulsar/pulsar-client-go/pulsar"
)
func main() {
//end of read stream
stop := 16
// Instantiate a Pulsar client
client, err := pulsar.NewClient(pulsar.ClientOptions{
URL: "pulsar://pulsar-broker.pulsar.svc.cluster.local:6650",
})
if err != nil {
log.Fatalf("Could not create client: %v", err)
}
// Use the client to instantiate a reader
reader, err := client.CreateReader(pulsar.ReaderOptions{
Topic: "ragnarok/transactions/requests",
StartMessageID: pulsar.EarliestMessage,
})
if err != nil {
log.Fatalf("Could not create reader: %v", err)
}
ctx := context.Background()
// Listen on the topic for incoming messages
cnt := 0
for {
msg, err := reader.Next(ctx)
if err != nil {
log.Fatalf("Error reading from topic: %v", err)
} else {
cnt++
content := string(msg.Payload())
fmt.Println(cnt, " -> ", msg.ID(), " -> ", content)
if cnt == stop {
fmt.Println("reached stream end. breaking out.")
defer reader.Close()
break
}
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tjiuming commented on issue #15045: reader-fd3917723f, 0] Failed to create consumer: BrokerPersistenceError
Posted by GitBox <gi...@apache.org>.
tjiuming commented on issue #15045:
URL: https://github.com/apache/pulsar/issues/15045#issuecomment-1090392708
The issue was fixed. I think the deep reason is unstable network, I noticed `Consumer#close` is triggered by `Channel#inactive`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org