You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/09/12 09:11:04 UTC
Slack digest for #general - 2018-09-12
2018-09-11 09:46:26 UTC - Daniele ASCIONE: @Daniele ASCIONE has joined the channel
----
2018-09-11 12:51:22 UTC - Mark Addison: In the Pulsar documentation there is an example of *Deploying Pulsar on Kubernetes*
`<https://pulsar.incubator.apache.org/docs/en/deploy-kubernetes/>`
In terms of deploying Pulsar in a virtualised/containerised environment where failures could be possible
Are there any thoughts from Pulsar developers/users concerning e.g.
• Backing up data (bookies) and restoration?
• Recovery from failure if the whole cluster were to fail?
Any thoughts would be welcome!
----
2018-09-11 12:56:04 UTC - Guillaume Pitel: Hi, I have an issue here : one of my broker stops sending messages to consumers.
----
2018-09-11 12:56:24 UTC - Guillaume Pitel:
----
2018-09-11 12:57:03 UTC - Guillaume Pitel: As you can see, MSG/s OUT of broker <http://pulsar-online-1.exensa.net|pulsar-online-1.exensa.net> is 0.00
----
2018-09-11 12:57:07 UTC - Matteo Merli: single topic or across multiple topics?
----
2018-09-11 12:57:12 UTC - Matteo Merli: (or partitions)
----
2018-09-11 12:57:27 UTC - Guillaume Pitel: multiple non-shared topics, one namespace
----
2018-09-11 12:57:40 UTC - Guillaume Pitel: wait, two namespaces actually
----
2018-09-11 12:58:05 UTC - Guillaume Pitel: no exception/error/warn in the logs apparently
----
2018-09-11 12:58:29 UTC - Guillaume Pitel: last time it happened, restarting the clients made the thing restart
----
2018-09-11 12:59:09 UTC - Matteo Merli: can you get the broker stats, with all the topics? From the broker machine: `bin/pulsar-admin broker-stats topics --indent`
----
2018-09-11 12:59:46 UTC - Matteo Merli: (also a jstack of the process might be useful)
----
2018-09-11 13:00:50 UTC - Guillaume Pitel: jstack of pulsar process ?
----
2018-09-11 13:02:34 UTC - Matteo Merli: yes
----
2018-09-11 13:03:12 UTC - Matteo Merli: (just to double check)
----
2018-09-11 15:03:32 UTC - Mark Addison: It would be good if responses could be made in this thread - to not interfere with other questions...!
----
2018-09-11 18:56:33 UTC - Nick Marchessault: Hi @Matteo Merli, I updated my java pulsar client version to the one above but am still seeing the 'discarding corruption errors'.. Does the pulsar infrastructure also need to be updated in order to avoid this error?
----
2018-09-11 18:57:03 UTC - Matteo Merli: Yes, that’s mostly on broker side
----
2018-09-12 08:07:57 UTC - Kesav Kolla: I've an use case where in I'm reading a message from queue work on the message. If my work logic raises any error I'm creating new message and placing the new message onto same queue and acknowledging original message. What I'm observing is lots of messages are going to backlog. I'm not able to understand is something wrong in my code or is it the behavior of backlog? In my code I create both consumer and producer with same subscription name to same queue. Would this be a problem?
Please some one help me I'm not able to understand the underlying reason for backlogs of messages
----
2018-09-12 08:12:44 UTC - Sijie Guo: @Kesav Kolla from the description, it seems to be okay. can you show a code example about how you do that?
----
2018-09-12 08:18:08 UTC - Kesav Kolla: @Sijie Guo Here is snippet of my python code
```
while True:
msg: _Message = self.jobsconsumer.receive()
try:
process(msg)
except:
self.logger.exception('Error in scraping job see logs for more details')
# Increase the retry count if needed and re-publish the message if RETRY_COUNT is not met
msgprops = msg.properties() if msg.properties() is not None else {}
retrycnt = msgprops['RETRY_COUNT'] if 'RETRY_COUNT' in msgprops else '0'
retrycnt = int(retrycnt.split('-')[-1])
if retrycnt < self.RETRY_COUNT:
self.jobsproducer.send('New message',
properties={'RETRY_COUNT': f'RETRY-{retrycnt+1}'})
else:
<http://self.logger.info|self.logger.info>('Exhausted all retries nothing can be done.')
self.jobsconsumer.acknowledge(msg)
```
----
2018-09-12 08:20:39 UTC - Kesav Kolla: is there anyway to bring the backlog messages to re-process them?
----
2018-09-12 08:36:56 UTC - Sijie Guo: @Kesav Kolla the code seems to be fine. can you check the topic stats and internal stats? it would be good to see if it is just because your application doesn’t process fast enough, or because the logic produces more messages.
```
bin/pulsar-admin topics stats <topic>
```
```
pulsar-admin topics stats-internal <topic>
```
----
2018-09-12 08:40:40 UTC - Kesav Kolla: stats
----
2018-09-12 08:41:42 UTC - Kesav Kolla:
----
2018-09-12 08:50:11 UTC - Sijie Guo: @Kesav Kolla the stats looks fine. but stats internal indicates some of the messages are not acknowledged (see gaps in “individuallyDeletedMessages”). Can you run stats-internal again, so we can compare individuallyDeletedMessages
----
2018-09-12 08:50:59 UTC - Kesav Kolla:
----
2018-09-12 08:59:36 UTC - Sijie Guo: (2469:3476..2469:3478], (2469:3479..2469:3480]
(2469:3476..2469:3478], (2469:3479..2469:3480]
it seems that some of the messages are never acked, for example, message `2469:3479`.
----
2018-09-12 09:00:11 UTC - Sijie Guo: what it your consumer settings?
----
2018-09-12 09:07:36 UTC - Kesav Kolla: I don't have any special settings in code. I just used `client.subscribe(topic, __name__, consumer_type=pulsar.ConsumerType.Shared`
----
2018-09-12 09:09:24 UTC - Kesav Kolla: is there any time in pulsar that it checks for ack? Because some times my process may take more time. The only time I don't send any ack when my I restart my code. That's the only code path the message is not acked in that case the client is closed so as the subscription. I'd expect message should be processed by another subscription which is alive
----