You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/05/27 23:22:35 UTC
[GitHub] [pulsar] irairdon opened a new issue #7059: Pulsar Producing Hangs when Kubernertes Node Restarts
irairdon opened a new issue #7059:
URL: https://github.com/apache/pulsar/issues/7059
When running pulsar on Kubernetes if you reboot a Kubernetes node any client producing will hand and be unable to produce new messages.
**To Reproduce**
Steps to reproduce the behavior:
1. Install helm chart for Pulsar on kubernetes
2. Use pulsar-perf tool to start producing messages on a single topic.
3. reboot one of the kubernetes nodes.
4. I have not found errors listed but the pulsar-perf tool will start showing 0 messages produced.
5. If you kill the broker pods one at a time eventually the producer will start producing messages again.
**Expected behavior**
I expect pulsar not to hang if a kuberentes node reboots.
**Desktop (please complete the following information):**
- Pulsar 2.5.2
- Kubernetes 1.13
**Additional context**
I haven't been able to reproduce be restarting individual services. It only occurs in a node restart. When the current producer is hung you can still create new topics and new producers that will not see the hang so it is related only to existing active topics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] larslorenzen edited a comment on issue #7059: Pulsar Producing Hangs when Kubernertes Node Restarts
Posted by GitBox <gi...@apache.org>.
larslorenzen edited a comment on issue #7059:
URL: https://github.com/apache/pulsar/issues/7059#issuecomment-648041771
Don't know if it is related but we had the same problem after a node restart.
The following logs came from the producer:
```
[persistent://mytenant/mynamespace/mytopic] [my-pod-5c5c89dfd4-c9rsn-mytopic] Failed to create producer: Producer is already present on the connection
[id: 0x0d32b91d, L:/1.2.3.4:51954 - R:pulsar-proxy.pulsar.svc.cluster.local/1.2.3.4:6650] Received error from server: Producer is already present on the connection
```
The problem was only resolved by restarting the producing pod.
Pulsar broker version: 2.5.2
Pulsar producer version: 2.5.2
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] larslorenzen commented on issue #7059: Pulsar Producing Hangs when Kubernertes Node Restarts
Posted by GitBox <gi...@apache.org>.
larslorenzen commented on issue #7059:
URL: https://github.com/apache/pulsar/issues/7059#issuecomment-648041771
Don't know if it is related but we had the same problem after a node restart.
The following logs came from the producer:
`[persistent://mytenant/mynamespace/mytopic] [my-pod-5c5c89dfd4-c9rsn-mytopic] Failed to create producer: Producer is already present on the connection`
The problem was only resolved by restarting the producing pod.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie closed issue #7059: Pulsar Producing Hangs when Kubernertes Node Restarts
Posted by GitBox <gi...@apache.org>.
sijie closed issue #7059:
URL: https://github.com/apache/pulsar/issues/7059
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7059: Pulsar Producing Hangs when Kubernertes Node Restarts
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7059:
URL: https://github.com/apache/pulsar/issues/7059#issuecomment-673325311
This issue has been fixed by #7401
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] larslorenzen edited a comment on issue #7059: Pulsar Producing Hangs when Kubernertes Node Restarts
Posted by GitBox <gi...@apache.org>.
larslorenzen edited a comment on issue #7059:
URL: https://github.com/apache/pulsar/issues/7059#issuecomment-648041771
Don't know if it is related but we had the same problem after a node restart.
The following logs came from the producer:
```
[persistent://mytenant/mynamespace/mytopic] [my-pod-5c5c89dfd4-c9rsn-mytopic] Failed to create producer: Producer is already present on the connection
[id: 0x0d32b91d, L:/100.96.4.11:51954 - R:pulsar-proxy.pulsar.svc.cluster.local/1.2.3.4:6650] Received error from server: Producer is already present on the connection
```
The problem was only resolved by restarting the producing pod.
Pulsar broker version: 2.5.2
Pulsar producer version: 2.5.2
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] larslorenzen edited a comment on issue #7059: Pulsar Producing Hangs when Kubernertes Node Restarts
Posted by GitBox <gi...@apache.org>.
larslorenzen edited a comment on issue #7059:
URL: https://github.com/apache/pulsar/issues/7059#issuecomment-648041771
Don't know if it is related but we had the same problem after a node restart.
The following logs came from the producer:
`[persistent://mytenant/mynamespace/mytopic] [my-pod-5c5c89dfd4-c9rsn-mytopic] Failed to create producer: Producer is already present on the connection`
The problem was only resolved by restarting the producing pod.
Pulsar broker version: 2.5.2
Pulsar producer version: 2.5.2
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7059: Pulsar Producing Hangs when Kubernertes Node Restarts
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7059:
URL: https://github.com/apache/pulsar/issues/7059#issuecomment-673324613
@larslorenzen Your issue is different. The "producer is already presented" is a race condition when producer disconnects and reconnects. The problem has been addressed in recent releases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org