You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/07/11 09:11:03 UTC
Slack digest for #general - 2018-07-11
2018-07-10 14:44:50 UTC - Josh West: any reason why i cannot assign a cluster to a tenant/namespace? i have that same cluster assigned to a namespace for another tenant. is it one cluster per tenant or something?
----
2018-07-10 14:50:25 UTC - Josh West: seems that you can only set the cluster(s) on namespace creation time, but later on `set-clusters` isn't working
----
2018-07-10 15:48:50 UTC - Nicolas Ha: Hello @Sijie Guo :slightly_smiling_face: Did you have time to look at this?
----
2018-07-10 16:26:39 UTC - Matteo Merli: What error is that giving? The `set-clusters` operation is supposed to be working for that exact case: changing the initial set of replication clusters
----
2018-07-10 16:46:32 UTC - Yuvaraj Loganathan: How can I set the schema compatibility level Like Forward , backward and both in pulsar ?
----
2018-07-10 16:47:03 UTC - Yuvaraj Loganathan: <https://pulsar.apache.org/docs/latest/getting-started/ConceptsAndArchitecture/#schema-types>
----
2018-07-10 16:47:28 UTC - Yuvaraj Loganathan: A schema already exists; the producer connects using a new schema that is compatible
----
2018-07-10 16:48:08 UTC - Yuvaraj Loganathan: What will happen if the schema already exists but the new schema is not compatible ?
----
2018-07-10 16:48:41 UTC - Matteo Merli: The producer creation will be rejected in that case
----
2018-07-10 16:54:49 UTC - Yuvaraj Loganathan: @Matteo Merli Thanks. On what basis the schema will be marked as invalid ? I am able think of cases like change in field type and missing fields (am I right here ?)? Adding an new field is considered as compatible ?
----
2018-07-10 16:59:27 UTC - Matteo Merli: Yes, for JSON it pretty much boils down to no-type changes. You can add/remove fields in compatible way. For 2.1 we’ve added Avro / Protobuf and both will use Avro’s own checks for schema validation
----
2018-07-10 17:01:56 UTC - Yuvaraj Loganathan: @Matteo Merli Excellent ! Seems like 2.1 will be an wonderful release :smile:
----
2018-07-10 17:02:49 UTC - Matteo Merli: :slightly_smiling_face: Yes! Working on getting that out of the door. Should be very close now.
bananadance : Yuvaraj Loganathan
+1 : Yuvaraj Loganathan
----
2018-07-10 18:53:42 UTC - Grant Wu: @Matteo Merli Uh, two things:
----
2018-07-10 18:54:14 UTC - Grant Wu: 1. I decided that I probably don’t need that feature in the Reader interface - we’re just going to use the Consumer interface and `resetCursor`, and push the work of stopping reading to our clients
----
2018-07-10 18:55:35 UTC - Grant Wu: Although - I guess it’s worth asking - what do you think about having a `start-timestamp` parameter in the Consumer interface? That would also bring a similar benefit of not needing to call the Admin API
----
2018-07-10 18:58:35 UTC - Grant Wu: 2. So, as I understand it, acknowledgement in the Reader interface (<https://pulsar.incubator.apache.org/docs/latest/clients/WebSocket/#Acknowledgingthemessage-3a2lsi15>) is not the same as consumer acknowledgement, and does not have the effects on brokers discarding messages that consumer acks have?
----
2018-07-10 19:04:16 UTC - Matteo Merli: the websocket interface is slightly different. Since there is no other way to perform “flow-control” we ended up using “ack” even for the websocket reader. If you look at Java interface (<http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/Reader.html>) (or C++/Python/Go) there is really no “ack” method there.
----
2018-07-10 19:05:25 UTC - Grant Wu: Hrm, do the client libraries not go over WS?
----
2018-07-10 19:06:41 UTC - Grant Wu: But okay, I see, that makes sense. I got the sense that Reader acks (in the WS interface) were purely for flow control
----
2018-07-10 19:06:57 UTC - Matteo Merli: No, WS is provided as a way to interact with Pulsar without using client library. Pulsar uses its own protocol for client->broker communication.
----
2018-07-10 19:07:59 UTC - Matteo Merli: Java and C++ both implement that protocol (with wrappers for Python and Go based on C++ library)
ok : Grant Wu, Ali Ahmed
----
2018-07-10 19:23:09 UTC - Grant Wu: what do you think about having a `start-timestamp` parameter in the Consumer interface?
----
2018-07-10 19:25:08 UTC - Grant Wu: @Matteo Merli
----
2018-07-10 19:38:02 UTC - Grant Wu: Or really what would be nice is
----
2018-07-10 19:38:49 UTC - Grant Wu: Just an endpoint that lets us lookup message ID with a timestamp
----
2018-07-10 19:52:08 UTC - Grant Wu: Wait: how does message retention interact with the consumer endpoint? If message retention is enabled, where does a new subscription get positioned?
----
2018-07-10 19:53:52 UTC - Grant Wu: Sorry for all these questions :sweat:
----
2018-07-10 20:04:15 UTC - Joe Francis: You should be able to. We do it all the time. Is the tenant provisioned on that cluster? What's the error?
----
2018-07-10 21:43:48 UTC - Grant Wu: <https://pulsar.incubator.apache.org/docs/latest/cookbooks/RetentionExpiry/> says “immediately delete all messages that have been acknowledged on every subscription”
<https://pulsar.incubator.apache.org/docs/latest/getting-started/ConceptsAndArchitecture/#Messageretentionandexpiry-iwgen> says “immediately delete all messages that have been acknowledged by a consumer”
----
2018-07-10 21:44:08 UTC - Grant Wu: I feel like “every subscription” and “a consumer” are… very different things
----
2018-07-10 21:51:40 UTC - Grant Wu: oh, I had leftover data from before.
----
2018-07-10 22:15:18 UTC - Grant Wu: I think I’ve empirically answered my question: “how does message retention interact with the consumer endpoint? If message retention is enabled, where does a new subscription get positioned?”
Answer is - at the beginning (first retained message)
----
2018-07-10 22:17:37 UTC - Sijie Guo: @Grant Wu when a new subscription join, by default it positions the subscription at the latest message, which is at the end; you can change the behavior by specifying SubscriptionInitialPosition to start from earliest message, which then is at the beginning (first retained message)
----
2018-07-10 22:18:15 UTC - Grant Wu: Hrm, that’s not the behavior i’m seeing
----
2018-07-10 22:18:28 UTC - Grant Wu: Is that with message retention on?
----
2018-07-10 22:19:00 UTC - Grant Wu: Here’s how I tested it:
----
2018-07-10 22:19:54 UTC - Grant Wu: `./pulsar standalone`
in window 1
`for i in {1..10000}; do ./pulsar-client produce TestTopic -m "Message $(date)"; sleep 8; done`
in window 2
`./pulsar-client consume -r 1 -n 0 -s Subscription1 TestTopic`
in window 3
`./pulsar-client consume -r 1 -n 0 -s Subscription2 TestTopic`
in window 4
----
2018-07-10 22:20:15 UTC - Grant Wu: Test routine was to start window 1, then window 3, then window 2
----
2018-07-10 22:20:22 UTC - Grant Wu: Verify that a few messages are produced and consumed.
----
2018-07-10 22:20:41 UTC - Grant Wu: Then start window 4. Note that the first message consumed in window 4 is not the first message consumed in window 3.
----
2018-07-10 22:20:59 UTC - Grant Wu: Then, run `./pulsar-admin namespaces set-retention --size -1 --time -1 public/default`
----
2018-07-10 22:21:23 UTC - Grant Wu: Repeat test routine
----
2018-07-10 22:21:31 UTC - Grant Wu: Note that window 4 now starts at the exact same message at window 3
----
2018-07-10 22:21:39 UTC - Grant Wu: (i.e. the beginning)
----
2018-07-10 22:21:59 UTC - Sijie Guo: do you delete subscriptions and topic data?
----
2018-07-10 22:22:13 UTC - Sijie Guo: before you repeat the test
----
2018-07-10 22:22:21 UTC - Grant Wu: I shut down all consumers/producers in between the tests; I let window 3 drain off old messages for a while in between tests
----
2018-07-10 22:22:40 UTC - Grant Wu: oh wait, no
----
2018-07-10 22:22:47 UTC - Grant Wu: you’re right, it’s saving the old subscription
----
2018-07-10 22:22:47 UTC - Grant Wu: ugh.
----
2018-07-10 22:23:38 UTC - Sijie Guo: you can use different topics and set-retention before you start window 4
----
2018-07-10 22:26:23 UTC - Grant Wu: Okay yes you’re right
----
2018-07-10 22:26:44 UTC - Grant Wu: sorry for the confusion
----
2018-07-10 22:27:51 UTC - Grant Wu: Is SubscriptionInitialPosition available in the WS interface? I don’t see it here <https://pulsar.incubator.apache.org/docs/latest/clients/WebSocket/#Consumerendpoint-yfa6po>
----
2018-07-10 23:29:05 UTC - Sijie Guo: SubscriptionInitialPosition was contributed at 2.0. it is probably not available in WS yet. we are working to bring the feature parity between languages in the following 2.1 and 2.2 cases.
----
2018-07-11 07:30:55 UTC - Idan: Morning guys, i have an issue on pulsar that i cant nail down
----
2018-07-11 07:31:58 UTC - Idan: iam getting unack warning from pulsar but still messages not being re-consumed. We set timeout val on pulsar to be 3 seconds. The timeouts doesnt make sense as by our logs we acking much less then 3 seconds
----
2018-07-11 07:32:01 UTC - Idan: 2018-07-11 07:17:49,953 WARN org.apache.pulsar.client.impl.UnAckedMessageTracker - [ConsumerBase{subscription=‘subscr-2’, consumerName=‘f77d0’, topic=‘<persistent://pulsar-pulsar-cluster/public/default/MyQueue’>}] 1 messages have timed-out
----
2018-07-11 07:32:22 UTC - Idan: iam getting this for every produce-consume message
----
2018-07-11 08:09:28 UTC - Sijie Guo: how did you ack?
----
2018-07-11 08:36:01 UTC - Idan: @Sijie Guo we just checked our logs we are acking after 150ms.
----
2018-07-11 08:36:04 UTC - Idan: how did I ack? sec
----
2018-07-11 08:36:28 UTC - Idan: consumer.acknowledgeAsync(messageId).handle((result, ex) -> {
if (ex == null) {
log.trace(“Message acked: {}“, messageId);
successHandler.accept(messageId);
} else {
log.error(“Error acking message to pulsar broker. errorMsg {} topic {} eventId {}“, new Object[]{ex, topic, eventId});
failHandler.accept((Exception)ex);
}
----
2018-07-11 08:36:54 UTC - Idan: the thing is that we dont see re-delivery so that warning is suspicies coz if we dont ack we suspect to see re-delivery
----
2018-07-11 08:37:12 UTC - Idan: i need to know if something is wrong on our side before we can proceed
----
2018-07-11 08:40:44 UTC - Idan: seems like we get it on all our pulsar topic consumers
----
2018-07-11 09:04:08 UTC - Idan: @Idan uploaded a file: <https://apache-pulsar.slack.com/files/UALJD8929/FBNG7SE6N/-.m|Untitled>
----