Slack digest for #general - 2019-12-14

[Apache Pulsar Slack <ap...@gmail.com>]

2019-12-13 11:48:47 UTC - Jens Fosgerau: 1. What happens to produced messages if there's no subscriptions? (by default)
2. Will new subscriptions to an existing topic start from earliest message, or only new ones created after creating the subscription?
3. Can individual subscriptions ack the same message?
Edit: I kind of answered 3. myself by just trying it out, and that seems like a definitive yes. Follow up question is what happens to old subscriptions?
----
2019-12-13 11:54:43 UTC - Penghui Li: 1. if no data retention there, the published messages will be removed.
2. By default, the position of the new subscription is the last published message, you can specify the initial message id which you like
3. Each subscription has a cursor and can read all messages of the topic, also can ack
----
2019-12-13 11:55:41 UTC - Jens Fosgerau: Is it feasible to have endless retention? (I mean, does it make sense from a performance perspective)
----
2019-12-13 11:59:10 UTC - Penghui Li: If you have enough disk to store the data , it is feasible
----
2019-12-13 12:00:01 UTC - Jens Fosgerau: Just got to figure out how to do it properly with the go client, in that case. Thanks :smile:
----
2019-12-13 12:01:26 UTC - Penghui Li: the retention policy can setting at broker side, you can use pulsar-admin set-retention
----
2019-12-13 12:01:43 UTC - Jens Fosgerau: I think we want it on a topic-by-topic basis.
----
2019-12-13 12:01:55 UTC - Jens Fosgerau: But that could potentially also be achieved on a namespace level.
----
2019-12-13 12:04:36 UTC - Penghui Li: the topic level policy is not ready now, but already underway
----
2019-12-13 12:05:02 UTC - Jens Fosgerau: It is on namespace level, right? At least that's how I read it in the docs.
----
2019-12-13 12:05:27 UTC - Penghui Li: yes on namespace level
----
2019-12-13 12:05:43 UTC - Jens Fosgerau: That'll be fine for our use-cases at the moment. :smile:
----
2019-12-13 12:05:53 UTC - Penghui Li: :+1:
----
2019-12-13 12:06:19 UTC - Jens Fosgerau: I think the only thing we really lack in the go library to get going, is dead letter topics.
----
2019-12-13 12:22:25 UTC - Jens Fosgerau: @Penghui Li I've enabled retention by size and time of `-1` for a topic without subscriptions.
I produced messages to a topic that didn't exist, so it was autocreated, but according to `pulsar-admin topics stats` the storageSize and backlogSize is still `0`.

Same thing goes for the pulsar-dashboard, but I assume they access the data in the same way.

Edit: when I say for a topic, I mean the default namespace.
----
2019-12-13 12:23:56 UTC - Penghui Li: Which version of pulsar are you using?
----
2019-12-13 12:24:28 UTC - Jens Fosgerau: ```<http://docker.io/apachepulsar/pulsar@sha256:d7e2d4b0fe795a091669fb1309585f65c3ce7d666110029023d8f01fde3922e3|docker.io/apachepulsar/pulsar@sha256:d7e2d4b0fe795a091669fb1309585f65c3ce7d666110029023d8f01fde3922e3>```
----
2019-12-13 12:24:30 UTC - Jens Fosgerau: aka 2.4.1
----
2019-12-13 12:24:38 UTC - Jens Fosgerau: But I am using the standalone variant
----
2019-12-13 12:29:51 UTC - Jens Fosgerau: I also couldn't get it to unsubscribe/delete topics using `pulsar-dashboard` and `pulsar-admin` so I decided to reset my data, but to no avail.
----
2019-12-13 12:30:57 UTC - Jens Fosgerau: Not to mention the pulsar-dashboard tenants-page is reporting wrong numbers of namespaces, topics, producers, subscriptions and consumers. It's really confusing :stuck_out_tongue:
----
2019-12-13 12:34:23 UTC - Jens Fosgerau: After creating a subscription, it seems like ended up setting storage size to ~80KB, but unsure if that's the correct amount
----
2019-12-13 12:56:00 UTC - Jens Fosgerau: After playing with it more, it just seems like the displayed storageSize and backlogSize when calling `topic stats` does not actually give correct information about how many messages are in the topic.

When there's no subscriptions it always seems to be 0. As soon as I add a subscription and read some of the messages, it has a 'high' value. As soon as all messages are ack'ed in the new subscription it goes to a lower number.
----
2019-12-13 12:56:06 UTC - Jens Fosgerau: Deleting all subscriptions takes it back to 0
----
2019-12-13 13:01:05 UTC - Penghui Li: The issue already fixed in pulsar 2.4.2 <https://github.com/apache/pulsar/pull/5108>

----
2019-12-13 13:01:35 UTC - Jens Fosgerau: Hmm. I wonder why I have an old version, as I recently installed what seemed to be the latest
----
2019-12-13 13:01:59 UTC - Jens Fosgerau: Oh.. it was released 4 days ago
----
2019-12-13 13:02:13 UTC - Penghui Li: Yes
----
2019-12-13 13:02:28 UTC - Jens Fosgerau: I'll swing the upgrade hammer, and see how it looks.
----
2019-12-13 13:04:16 UTC - Penghui Li: :+1:
----
2019-12-13 13:04:32 UTC - Jens Fosgerau: Upgrading the pulsar-dashboard image broke some postgres script it seems. Edit: <https://github.com/apache/pulsar/issues/5847> - Ah
----
2019-12-13 13:08:19 UTC - Penghui Li: I think you can upgrade the broker only, since the issue is related to broker not dashboard
----
2019-12-13 13:11:21 UTC - Jens Fosgerau: storageSize is now correct, however the backlogSize is always 0 without subscriptions.
----
2019-12-13 13:11:42 UTC - Jens Fosgerau: As soon as I add a subscription, it goes to a super high number, and eventually drops to a lower number when it's done Acking all messages
----
2019-12-13 13:13:14 UTC - Penghui Li: If you do not have subscription, the backlogSize should be 0
----
2019-12-13 13:13:38 UTC - Jens Fosgerau: And with subscriptions, shouldn't it eventually go back to 0, if all subscriptions are done ack'ing?
----
2019-12-13 13:14:06 UTC - Jens Fosgerau: ```{
  "msgRateIn" : 0.0,
  "msgThroughputIn" : 0.0,
  "msgRateOut" : 0.0,
  "msgThroughputOut" : 0.0,
  "averageMsgSize" : 0.0,
  "storageSize" : 28999143,
  "backlogSize" : 411423,
  "publishers" : [ ],
  "subscriptions" : { 
    "testqueue2-test" : {
      "msgRateOut" : 0.0,
      "msgThroughputOut" : 0.0,
      "msgRateRedeliver" : 0.0,
      "msgBacklog" : 0,
      "blockedSubscriptionOnUnackedMsgs" : false,
      "msgDelayed" : 0,
      "unackedMessages" : 0,
      "type" : "Shared",
      "msgRateExpired" : 0.0,
      "consumers" : [ ],
      "isReplicated" : false
    }
  },
  "replication" : { },
  "deduplicationStatus" : "Disabled"
}```
----
2019-12-13 13:14:24 UTC - Penghui Li: Yes if all subscriptions are acked all messages, it should be 0
----
2019-12-13 13:14:30 UTC - Jens Fosgerau: But it's not :disappointed:
----
2019-12-13 13:14:52 UTC - Jens Fosgerau: It's 0 for the subscription, but has a higher number total on the topic
----
2019-12-13 13:16:17 UTC - Penghui Li: I see, can you help add an issue at github? contributors can help fix the problem. you’d better add your reproduce steps
----
2019-12-13 13:16:45 UTC - Jens Fosgerau: I'll have to make it more reproducible I guess.
----
2019-12-13 13:17:01 UTC - Penghui Li: :+1:
----
2019-12-13 13:17:21 UTC - Jens Fosgerau: Right now I just had a go producer write as many messages as it could for 30 seconds, and then created and deleted the same subscription using a go consumer from earliest message.
----
2019-12-13 16:29:53 UTC - Brian Doran: @Sijie Guo Just a note here, I've tried multiple things:
• disabled fsync (little to no difference)
One thing I've noticed it that spread of topics is quite uneven across the 3 nodes. Also that 1 node is particularly busy with the bulk of the in-rate.
----
2019-12-13 17:45:36 UTC - Brian Doran: in fact the in-rate is 0.0 on 2 nodes in my 3 node cluster
----
2019-12-13 20:25:43 UTC - Rafael de Andrade Sousa: @Rafael de Andrade Sousa has joined the channel
----
2019-12-14 04:40:01 UTC - Sijie Guo: • are these 16 disks raided? or are you using them in JBOD way?
• how many topics are you used for testing?
• how did you do the test? Are you using `pulsar-perf`?
----