You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/01/23 09:11:03 UTC
Slack digest for #general - 2020-01-23
2020-01-22 11:30:50 UTC - Swaroop Kumar: Example scenario:
user : u1 -> 10 messages
user : u2 -> 1 message
There are 3 consumers (c1, c2, c3) and my limit per user is 2. So when c1 and c2 takes 2 messages of u1.
c3 should take a message of u2 but not of u1.
----
2020-01-22 11:31:10 UTC - Swaroop Kumar: Example scenario:
user : u1 -> 10 messages
user : u2 -> 1 message
There are 3 consumers (c1, c2, c3) and my limit per user is 2. So when c1 and c2 takes 2 messages of u1.
c3 should take a message of u2 but not of u1.
----
2020-01-22 13:59:03 UTC - Devin G. Bost: You're going to need a place to store the state.
Is there a reason that you're using consumers instead of Pulsar Functions?
Regarding storing the state, you have a few options. You could use state storage in Pulsar functions (which is still in preview), you could use a stream processing engine like Apache Flink, or you could use hot storage like Apache Ignite. If you use Ignite, you can also easily persist your state to disk. In any case, Pulsar functions will work unless there's more to your use case that I'm missing.
----
2020-01-22 15:01:23 UTC - Alexandre DUVAL: Hi, in pulsar v2.5.0 the topic storage size is now containing the "real" internal stats in bookies?
----
2020-01-22 15:23:49 UTC - Penghui Li: What does “real” internal stats mean, i just remember this PR <https://github.com/apache/pulsar/pull/5108> has introduced some changes related to the storage size.
----
2020-01-22 15:24:10 UTC - Penghui Li: But this PR is already in the 2.4.2 release
----
2020-01-22 15:37:57 UTC - Alexandre DUVAL: Ok, I was running v2.4.0 so that's it
----
2020-01-22 15:38:02 UTC - Alexandre DUVAL: ty
----
2020-01-22 15:40:25 UTC - Penghui Li: Ok, v2.4.0 use the backlog size as the storage size, it was fixed after v2.4.2.
----
2020-01-22 15:45:05 UTC - David Kjerrumgaard: I can't think of an easy way to do that other than having each consumer keep an internal counter of messages processed. Every time a consumer takes the message they increment the counter and check its value. If it equals 3 (in this case), they send a negative ack to reject the message and reset the counter. The negatively acked message will be delivered to another consumer in the group.
----
2020-01-22 16:13:50 UTC - Guilherme Perinazzo: Is there a guide for migrating to 2.5?
+1 : Fernando
----
2020-01-22 16:35:09 UTC - Alexandre DUVAL: yup, i asked to ensure but i'm right thx
----
2020-01-22 16:35:33 UTC - Alexandre DUVAL: Same as other versions: <https://pulsar.apache.org/docs/en/administration-upgrade/>
----
2020-01-22 16:53:50 UTC - Tarek Shaar: Does anyone know if Apache Pulsar runs on open JDK?
----
2020-01-22 16:54:16 UTC - Matteo Merli: Yes, it's does
----
2020-01-22 16:55:11 UTC - rmb: Hi all, I'm wondering about deduplication. In particular, the instructions at <https://pulsar.apache.org/docs/en/next/cookbooks-deduplication/> say that if a broker has deduplication enabled, producers should set timeouts to infinity. Why is that? Is the issue that if a send() times out but the broker ultimately receives the message, the producer might resend the message with a different sequenceId and break deduplication? If my program set a timeout and also set sequenceIds manually, would everything work?
----
2020-01-22 18:24:02 UTC - Nikita Mathur: @Nikita Mathur has joined the channel
----
2020-01-23 00:02:36 UTC - Rohit Pal: @Rohit Pal has joined the channel
----
2020-01-23 01:09:16 UTC - David Kjerrumgaard: The documentation for resource-quotas isn't very clear, can some elaborate on what they are? At what level are they applied, i.e on a per-namespace level, etc.
----
2020-01-23 01:09:19 UTC - David Kjerrumgaard: ./pulsar-admin resource-quotas get
{
"msgRateIn" : 40.0,
"msgRateOut" : 120.0,
"bandwidthIn" : 100000.0,
"bandwidthOut" : 300000.0,
"memory" : 80.0,
"dynamic" : true
}
----
2020-01-23 01:11:09 UTC - David Kjerrumgaard: How does one interpret these values? `msgRateIn: 40` What does that mean? 40 msg per second ? Is that per topic? Thanks
----