You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/04/11 09:11:02 UTC
Slack digest for #general - 2018-04-11
2018-04-10 13:52:37 UTC - Karthik Palanivelu: @Sijie Guo Thanks for clarifying on the global use cases and upcoming 2.0. This would help me move on. Are you referring to redeliverUnacknowledgedMessages() or Can you please let me know the setting to have my consumer to read from the earliest message?
----
2018-04-10 14:31:56 UTC - Karthik Palanivelu: Hello All, Any road map to develop kerberos authentication?
----
2018-04-10 17:10:30 UTC - Kishore Karunakaran: @Kishore Karunakaran has joined the channel
----
2018-04-10 18:48:51 UTC - Sijie Guo: @Karthikeyan Palanivelu
I was referring - SubscriptionIntialPosition : <https://pulsar.incubator.apache.org/api/client/org/apache/pulsar/client/api/SubscriptionInitialPosition.html>
<https://pulsar.incubator.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#subscriptionInitialPosition-org.apache.pulsar.client.api.SubscriptionInitialPosition->
I think it is coming at 2.0 release.
or you can use reset-cursor command in pulsar-admin for 1.x versions
<https://pulsar.incubator.apache.org/docs/latest/admin-api/persistent-topics/#Resetcursor-ci00ao>
----
2018-04-10 18:52:57 UTC - Sijie Guo: @Karthikeyan Palanivelu - re “kerberos authentication” - currently there is kerberos authentication available between brokers and bookies. but we don’t have kerberos auth between client and brokers (currently only TLS and Authenz are supported). we will probably look into kerberos authentication in 2.1.
----
2018-04-10 18:54:48 UTC - Sanjeev Kulkarni: @Karthikeyan Palanivelu contributions from community also welcome :slightly_smiling_face:
----
2018-04-10 19:10:33 UTC - Karthik Palanivelu: Sure Sanjeev, I am making myself comfortable. Let me look into it
----
2018-04-10 19:12:06 UTC - Karthik Palanivelu: @Sijie Guo Thanks for your update. I tried with MessageId.earliest - brought all message from the start as you mentioned. I need only unacknowledged messages - Please advice.
----
2018-04-10 19:13:29 UTC - Karthik Palanivelu: I tried redeliverUnacknowledgedMessages on Consumer eventually ended up producing 2 messages for same ID as the method suggest. We need only unacknowledged messages - May be a new method?
----
2018-04-10 19:14:15 UTC - Matteo Merli: > I tried redeliverUnacknowledgedMessages on Consumer eventually ended up producing 2 messages for same ID as the method suggest
Can you clarify this?
----
2018-04-10 19:14:57 UTC - Matteo Merli: `redeliverUnacknowledgedMessages()` will trigger the re-delivery of messages that were “received” by the application but not acknowledged yet
----
2018-04-10 19:15:29 UTC - Matteo Merli: it can be automatically triggered based on time (`ConsumerConfiguration.setAckTimeout()`)
----
2018-04-10 19:20:39 UTC - Karthik Palanivelu: @Matteo Merli I missed the logs, Let me get the logs. Till that time please ignore
----
2018-04-10 19:21:46 UTC - Matteo Merli: > Hello all, Can you please let me know how I can consume all the messages that are not consumed yet? For example, I produced x set of messages with no consumers. After producing messages, when client comes online as Shared/Exclusive not consuming the messages. It consumes messages that were produced after consumer is live. Please let me know if I am missing any.
So if you created the subscription initially, the messages would have been retained
----
2018-04-10 19:22:06 UTC - Matteo Merli: I’m not sure if that was your question
----
2018-04-10 19:25:54 UTC - Karthik Palanivelu: My scenario is in case if there is no consumer when producer producing messages. After a while a consumer subscribes to the topic - I need to consume all the messages that were produced before which were not acknowledged.
----
2018-04-10 19:30:08 UTC - Piotr: You´d have to configure the namespace/topic to have unlimited retention, and then start a subscription from the first message I guess?
----
2018-04-10 19:30:59 UTC - Matteo Merli: @Karthikeyan Palanivelu That’s the default behavior for consumer/subscription. You just need to have created the subscription upfront
----
2018-04-10 19:31:48 UTC - Matteo Merli: the subscription is created the first time a consumer tries to use it (and it’s persisted when the consumer disconnects)
----
2018-04-10 19:32:08 UTC - Matteo Merli: otherwise you can use REST API or CLI tools to create a subscription
----
2018-04-10 19:40:51 UTC - Karthik Palanivelu: @Matteo Merli This is tricky lets try to explain here. I produced a message. No Consumers. Then I added a consumer with my-file-sub-1 as subscription name. No message was consumed. As you said. I disconnected the Consumer. Enabled the consumer.redeliverUnacknowledgedMessages(). Restarted the consumer, no message consumed. Disconnected the consumer. Produced another message. Started the consumer as the earlier state, consumes messages twice. Please find below the logs - same message Id:
----
2018-04-10 19:42:31 UTC - Karthik Palanivelu: @Karthik Palanivelu uploaded a file: <https://apache-pulsar.slack.com/files/U7VRE0Q1G/FA483JKFE/-.txt|Untitled>
----
2018-04-10 19:43:32 UTC - Karthik Palanivelu: If the consumer is alive, I get only one time delivery.
----
2018-04-10 19:44:42 UTC - Sijie Guo: @Karthikeyan Palanivelu
“Then I added a consumer with my-file-sub-1 as subscription name. No message was consumed. As you said. I disconnected the Consumer. Enabled the consumer.redeliverUnacknowledgedMessages(). Restarted the consumer, no message consumed. Disconnected the consumer. Produced another message. Started the consumer as the earlier state, consumes messages twice. ”
do you have any code example about this sequence?
----
2018-04-10 19:45:00 UTC - Karthik Palanivelu: Yup
----
2018-04-10 19:45:04 UTC - Karthik Palanivelu: Let me share it
----
2018-04-10 19:45:35 UTC - Sijie Guo: cool thank you
----
2018-04-10 19:45:55 UTC - Karthik Palanivelu: @Karthik Palanivelu uploaded a file: <https://apache-pulsar.slack.com/files/U7VRE0Q1G/FA4C04E9H/-.txt|Untitled>
----
2018-04-10 19:54:20 UTC - Sijie Guo: @Karthikeyan Palanivelu
I see. let me try to explain to see if that makes sense or not.
so when you call “client.subscribe(topic,“my-file-sub-1”, config);“, the broker will try to start develivering “unacked” messages when subscribe call is done.
then you call “consumer.redeliverUnacknowledgedMessages();“, this would tell broker to redelivering “unacked” messages again.
so effectively you will receive two same messages. because you ask delivering “unacked” messages after subscribe and before you receive-and-ack.
----
back to your original question, if you want a “consumer”/subscription to receive messages, you need the subscription created prior to messages being produced.
in 2.0 release, you can do this:
Consumer consumer = client.newConsumer().subscriptionInitialPosition(SubscriptionInitialPosition.earliest).create();
while (true) {
// receive the messages.
}
in 1.x releases, you have to create the subscription prior to messages being produced.
hope this explains your questions
----
2018-04-10 20:47:22 UTC - Daniel Morgan: so maybe stupid question but I've heard pulsar supports kafka api does that mean I could use a co kafka plugin to connect to pulsar?
----
2018-04-10 20:56:12 UTC - Matteo Merli: > co kafka plugin
You mean a Kafka connect? It might work or not. Some of the plugins are relying on internal Kafka behaviors or break when topic names are in different format.
----
2018-04-10 21:09:23 UTC - Daniel Morgan: sorry a go client
----
2018-04-10 21:09:48 UTC - Daniel Morgan: confluent-go-kafka or the shopify one
----
2018-04-10 21:10:33 UTC - Matteo Merli: Oh, no the wrapper is only emulating the Kafka Java API at this point
----
2018-04-10 21:11:34 UTC - Matteo Merli: we plan to add Go client for Pulsar very soon. At that point it should be easy to add Kafka compat layer
----
2018-04-10 21:12:04 UTC - Karthik Palanivelu: @Sijie Guo Thanks for your detailed explanation
----
2018-04-10 21:22:35 UTC - Daniel Morgan: cheers @Matteo Merli
----