You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "KannarFr (via GitHub)" <gi...@apache.org> on 2023/06/07 11:27:21 UTC
[GitHub] [pulsar] KannarFr opened a new issue, #20526: [Bug] Broker sees topic fenced
KannarFr opened a new issue, #20526:
URL: https://github.com/apache/pulsar/issues/20526
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
2.11.1
### Minimal reproduce step
I have a cluster with thousands of topics and one became fenced see broker's logs:
```
Jun 07 11:20:48 yo-pulsar-broker-c3-n4 pulsar[336]: 2023-06-07T11:20:48,089+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.AbstractTopic - [persistent://tenant/ns/topic-partition-0] Attempting to add producer to a fenced topic
Jun 07 11:20:48 yo-pulsar-broker-c3-n4 pulsar[336]: 2023-06-07T11:20:48,089+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.pulsar.broker.service.ServerCnx - [/192.168.1.3:46348] Failed to add producer to topic persistent://tenant/ns/topic-partition-0: producerId=360, Topic is tempo
rarily unavailable
Jun 07 11:20:48 yo-pulsar-broker-c3-n4 pulsar[336]: 2023-06-07T11:20:48,102+0000 [BookKeeperClientWorker-OrderedExecutor-10-0] WARN org.apache.pulsar.broker.service.AbstractTopic - [persistent://tenant/ns/topic-partition-0] Attempting to add producer to a fenced topic
Jun 07 11:20:48 yo-pulsar-broker-c3-n4 pulsar[336]: 2023-06-07T11:20:48,102+0000 [BookKeeperClientWorker-OrderedExecutor-10-0] WARN org.apache.pulsar.broker.service.ServerCnx - [/192.168.1.3:46348] Failed to add producer to topic persistent://tenant/ns/topic-partition-0: producerId=361, Topic is temp
orarily unavailable
Jun 07 11:20:48 yo-pulsar-broker-c3-n4 pulsar[336]: 2023-06-07T11:20:48,112+0000 [BookKeeperClientWorker-OrderedExecutor-10-0] WARN org.apache.pulsar.broker.service.AbstractTopic - [persistent://tenant/ns/topic-partition-0] Attempting to add producer to a fenced topic
Jun 07 11:20:48 yo-pulsar-broker-c3-n4 pulsar[336]: 2023-06-07T11:20:48,112+0000 [BookKeeperClientWorker-OrderedExecutor-10-0] WARN org.apache.pulsar.broker.service.ServerCnx - [/192.168.1.3:46348] Failed to add producer to topic persistent://tenant/ns/topic-partition-0: producerId=362, Topic is temp
orarily unavailable
```
The topic was fenced for 30mins. I just restarted the broker and everything looks good now. Any idea? A wrong cache of fenced or something?
### What did you expect to see?
Not fenced
### What did you see instead?
Fenced
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Broker sees topic fenced [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #20526:
URL: https://github.com/apache/pulsar/issues/20526#issuecomment-1887767196
> @lhotari Thanks for pointing out to #20540
>
> We're using currently v3.1.1, but we will get the chance to test with #20540 in a few days (cc @KannarFr )
@StevenLeRoux @KannarFr FYI, there's a new bug report #21860 in this area with a promising bug fix in the Bookkeeper client in the works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Broker sees topic fenced [pulsar]
Posted by "KannarFr (via GitHub)" <gi...@apache.org>.
KannarFr closed issue #20526: [Bug] Broker sees topic fenced
URL: https://github.com/apache/pulsar/issues/20526
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Broker sees topic fenced [pulsar]
Posted by "ThomasVerhoeven1998 (via GitHub)" <gi...@apache.org>.
ThomasVerhoeven1998 commented on issue #20526:
URL: https://github.com/apache/pulsar/issues/20526#issuecomment-3196739611
> There's an ugly workaround for the problem: By setting `topicFencingTimeoutSeconds=5` for brokers, it will release the "fencing" after 5 seconds. However, there is a chance that this causes other problems such as data consistency problems. If metadata gets overwritten, it could lead to data loss.
We have set it to:
```
'topicFencingTimeoutSeconds':
value: '80'
```
But it doesn't resolve the issue when a topic gets fenced
We still get warnings like
```
[persistent://<namespace>/<topic>] [<pulsar-instance>]
Failed to create producer: {
"errorMsg": "org.apache.pulsar.broker.service.BrokerServiceException$TopicFencedException: Topic is temporarily unavailable",
"reqId": <redacted>,
"remote": "<broker-host>/<redacted-ip>:6651",
"local": "<client-host>/<redacted-ip>:<redacted-port>"
}
```
and
```
[persistent://<namespace>/<topic>] [<reader-id>]
Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException$LookupException: {
"errorMsg": "Topic is temporarily unavailable",
"reqId": <redacted>,
"remote": "<broker-host>/<redacted-ip>:6651",
"local": "<client-host>/<redacted-ip>:<redacted-port>"
}
```
(for your info I sanitized the errors)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Broker sees topic fenced [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #20526:
URL: https://github.com/apache/pulsar/issues/20526#issuecomment-1869464013
> Still impacted with this issue. Restarting brokers all day long doesn't seem a proper situation. I wonder if there is any production deployment that's not concerned by this issue, if so, how?
@StevenLeRoux which Pulsar version are you using? do you have a chance to test #20540 with a custom build?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #20526: [Bug] Broker sees topic fenced
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #20526:
URL: https://github.com/apache/pulsar/issues/20526#issuecomment-1633431328
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] KannarFr commented on issue #20526: [Bug] Broker sees topic fenced
Posted by "KannarFr (via GitHub)" <gi...@apache.org>.
KannarFr commented on issue #20526:
URL: https://github.com/apache/pulsar/issues/20526#issuecomment-1587666802
Unfortunately, I do not have suck logs retention. I should take a dump, my bad.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Broker sees topic fenced [pulsar]
Posted by "KannarFr (via GitHub)" <gi...@apache.org>.
KannarFr commented on issue #20526:
URL: https://github.com/apache/pulsar/issues/20526#issuecomment-1891870267
Thanks for pinging us @lhotari.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on issue #20526: [Bug] Broker sees topic fenced
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #20526:
URL: https://github.com/apache/pulsar/issues/20526#issuecomment-1581958101
Thanks for reporting this. This is an issue which has stayed unresolved for many years.
Some previous reports: #5284 and #14941.
There's an ugly workaround for the problem:
By setting `topicFencingTimeoutSeconds=5` for brokers, it will release the "fencing" after 5 seconds.
However, there is a chance that this causes other problems such as data consistency problems. If metadata gets overwritten, it could lead to data loss.
The recently merged fixes #18688 and #20527 could help improve the situation. I happened to investigate problems in this area yesterday.
* I noticed that I had forgotten to merge #18688, that is now merged.
* A previous solution to prevent some race conditions, #17526 was not effective any more.
* The PR #20527 fixes that. That is also merged.
I have created #20540 to address some issues that I have observed in the current solution. One of the remaining challenges in the PR is adding proper test coverage. I'm also waiting for feedback from other code contributors on the PR before finishing it. I'd appreciate feedback on the PR #20540.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Broker sees topic fenced [pulsar]
Posted by "StevenLeRoux (via GitHub)" <gi...@apache.org>.
StevenLeRoux commented on issue #20526:
URL: https://github.com/apache/pulsar/issues/20526#issuecomment-1869442231
Still impacted with this issue. Restarting brokers all day long doesn't seem a proper situation. I wonder if there is any production deployment that's not concerned by this issue, if so, how?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Broker sees topic fenced [pulsar]
Posted by "StevenLeRoux (via GitHub)" <gi...@apache.org>.
StevenLeRoux commented on issue #20526:
URL: https://github.com/apache/pulsar/issues/20526#issuecomment-1869643248
@lhotari Thanks for pointing out to #20540
We're using currently v3.1.1, but we will get the chance to test with #20540 in a few days (cc @KannarFr )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on issue #20526: [Bug] Broker sees topic fenced
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on issue #20526:
URL: https://github.com/apache/pulsar/issues/20526#issuecomment-1587661863
@KannarFr
> Jun 07 11:20:48 yo-pulsar-broker-c3-n4 pulsar[336]: 2023-06-07T11:20:48,112+0000 [BookKeeperClientWorker-OrderedExecutor-10-0] WARN org.apache.pulsar.broker.service.AbstractTopic - [persistent://tenant/ns/topic-partition-0] Attempting to add producer to a fenced topic
At this time, do you know if there is a bundle unload or a namespace unload executed?
You can check the HTTP request log to confirm it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org