You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "ragaur-tibco (via GitHub)" <gi...@apache.org> on 2024/03/29 07:07:53 UTC
[D] async modes in pulsar producer and consumer [pulsar]
GitHub user ragaur-tibco created a discussion: async modes in pulsar producer and consumer
What is the best way to check and confirm that Async mode is working in Pulsar producer and consumer
GitHub link: https://github.com/apache/pulsar/discussions/22382
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
GitHub user lhotari edited a comment on the discussion: async/sync mode in pulsar producer and consumer
@ragaur-tibco One more clarification needed: please explain what "working" means to you when you say "does async and sync mode work and how can I verify that the mode selected is working"?
Do you want to understand how to ensure that a message is successfully accepted for delivery by the broker?
Perhaps it would help to describe your use case. What are you trying to achieve?
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8980760
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
GitHub user lhotari added a comment to the discussion: async/sync mode in pulsar producer and consumer
@ragaur-tibco I'm really sorry, but you would have to explain to me, what "verify the mode I have chosen" means. Please elaborate on what you are verifying. :)
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8981118
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "ragaur-tibco (via GitHub)" <gi...@apache.org>.
GitHub user ragaur-tibco added a comment to the discussion: async/sync mode in pulsar producer and consumer
@lhotari Please advise how to verify the mode I have chosen, pulsar using the same mode to send messages from the producer and to receive messages from the consumer.
NOTE:
In my previous comment I mentioned that send() method internally calling the sendAsync()
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8980869
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
GitHub user lhotari edited a comment on the discussion: async/sync mode in pulsar producer and consumer
> 1. For synchronous mode we use send() method
> 2. For asynchronous mode we use sendAsync() method
@ragaur-tibco yes.
> if answer to the above two points is yes then why is send() method internally calling the sendAsync() method
@ragaur-tibco I don't really understand the point of the question. Do you have a problem with this implementation? What are your expectations for send and sendAsync? Is there some expectation that isn't met?
With the current implementation in the Pulsar Java client, it simply fulfills the requirements when send() internally calls sendAsync() method. Perhaps you have different expectations or requirements.
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8982733
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
GitHub user lhotari added a comment to the discussion: async/sync mode in pulsar producer and consumer
> 1. For synchronous mode we use send() method
> 2. For asynchronous mode we use sendAsync() method
@ragaur-tibco yes.
> if answer to the above two points is yes then why is send() method internally calling the sendAsync() method
@ragaur-tibco I don't really understand the point of the question. Do you have a problem with this implementation? What are your expectations for send and sendAsync? Is there some expectation that isn't met?
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8982733
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async mode in pulsar producer and consumer [pulsar]
Posted by "ragaur-tibco (via GitHub)" <gi...@apache.org>.
GitHub user ragaur-tibco edited a discussion: async mode in pulsar producer and consumer
What is the best way to check and confirm that Async mode is working in Pulsar producer and consumer
GitHub link: https://github.com/apache/pulsar/discussions/22382
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
GitHub user lhotari edited a comment on the discussion: async/sync mode in pulsar producer and consumer
> 1. For synchronous mode we use send() method
> 2. For asynchronous mode we use sendAsync() method
@ragaur-tibco yes.
> if answer to the above two points is yes then why is send() method internally calling the sendAsync() method
@ragaur-tibco I don't really understand the point of the question. Do you have a problem with this implementation? What are your expectations for send and sendAsync? Is there some expectation that isn't met?
With the current mode in the Pulsar Java client, it simply fulfills the requirements when send() internally calls sendAsync() method. Perhaps you have different expectations or requirements.
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8982733
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "ragaur-tibco (via GitHub)" <gi...@apache.org>.
GitHub user ragaur-tibco added a comment to the discussion: async/sync mode in pulsar producer and consumer
@lhotari I have tried very simple implementation with pulsar as a way to identify the workings of sync and async modes which lets to no results.
From the sources jars I noticed that `'send()'` method uses '`sendAsync()`' method so can you please confirm with the `'send()'` method that are we using sync mode or async mode
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8994098
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "visortelle (via GitHub)" <gi...@apache.org>.
GitHub user visortelle added a comment to the discussion: async/sync mode in pulsar producer and consumer
@ragaur-tibco I read the whole discussion 3 times and am still unsure I understood your question.
> I don't have any expectations or requirements I just want to know **how to test the both modes** any practical approach and anything in Pulsar SDK logs so that I can understand the internal process
@lhotari provided the file full of test cases: https://github.com/apache/pulsar/blob/master/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
I'm sure you'll find the answer to your question here.
Closing for now.
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8994539
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "ragaur-tibco (via GitHub)" <gi...@apache.org>.
GitHub user ragaur-tibco added a comment to the discussion: async/sync mode in pulsar producer and consumer
@lhotari In Apache Pulsar, does async and sync mode work and how can I verify that the mode selected is working?
There are multiple APIs used by the producer to send messages.
1. send()
2. sendAsync()
While internally, TypedMessageBuilderImpl.class present in **package org.apache.pulsar.client.impl** send() method is calling sendAsync()
```
public MessageId send() throws PulsarClientException {
try {
// enqueue the message to the buffer
CompletableFuture<MessageId> sendFuture = sendAsync();
if (!sendFuture.isDone()) {
// the send request wasn't completed yet (e.g. not failing at enqueuing), then attempt to triggerFlush
// it out
producer.triggerFlush();
}
return sendFuture.get();
} catch (Exception e) {
throw PulsarClientException.unwrap(e);
}
}
```
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8980638
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "ragaur-tibco (via GitHub)" <gi...@apache.org>.
GitHub user ragaur-tibco edited a discussion: async/sync mode in pulsar producer and consumer
What is the best way to check and confirm that Async mode is working in Pulsar producer and consumer
GitHub link: https://github.com/apache/pulsar/discussions/22382
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
GitHub user lhotari added a comment to the discussion: async/sync mode in pulsar producer and consumer
@ragaur-tibco I don't really understand the question. Yes, there are synchronous and asynchronous modes. What is it what you want to determine or verify?
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8981783
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
GitHub user lhotari edited a comment on the discussion: async/sync mode in pulsar producer and consumer
> 1. For synchronous mode we use send() method
> 2. For asynchronous mode we use sendAsync() method
@ragaur-tibco yes.
> if answer to the above two points is yes then why is send() method internally calling the sendAsync() method
@ragaur-tibco I don't really understand the point of the question. Do you have a problem with this implementation? What are your expectations for send and sendAsync? Is there some expectation that isn't met?
With the current mode in the Pulsar Java client, it simply fulfills the requirements when send() internally calls sendAsync() method.
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8982733
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "ragaur-tibco (via GitHub)" <gi...@apache.org>.
GitHub user ragaur-tibco edited a comment on the discussion: async/sync mode in pulsar producer and consumer
@lhotari
Can you please confirm below points, In Pulsar Java
1. For synchronous mode we use send() method
2. For asynchronous mode we use sendAsync() method
if answer to the above two points is yes then why is send() method internally calling the sendAsync() method
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8982290
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
GitHub user lhotari added a comment to the discussion: async/sync mode in pulsar producer and consumer
> What is the best way to check and confirm that Async/Sync mode is working in Pulsar producer and consumer
@ragaur-tibco what exactly do you mean with "async/sync mode working"? Please elaborate.
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8979066
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "ragaur-tibco (via GitHub)" <gi...@apache.org>.
GitHub user ragaur-tibco added a comment to the discussion: async/sync mode in pulsar producer and consumer
@lhotari I don't have any expectations or requirements I just want to know how to test the both modes any practical approach and anything in Pulsar SDK logs so that I can understand the internal process
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8991773
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "ragaur-tibco (via GitHub)" <gi...@apache.org>.
GitHub user ragaur-tibco added a comment to the discussion: async/sync mode in pulsar producer and consumer
@lhotari
Can you please confirm below points
1. For synchronous mode we use send() method
2. For asynchronous mode we use sendAsync() method
if answer to the above two points is yes then why is send() method internally calling the sendAsync() method
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8982290
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "ragaur-tibco (via GitHub)" <gi...@apache.org>.
GitHub user ragaur-tibco edited a discussion: async/sync mode in pulsar producer and consumer
What is the best way to check and confirm that Async/Sync mode is working in Pulsar producer and consumer
GitHub link: https://github.com/apache/pulsar/discussions/22382
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
GitHub user lhotari closed the discussion with a comment: async/sync mode in pulsar producer and consumer
> I have tried very simple implementation with pulsar as a way to identify the workings of sync and async modes which lets to no results.
It might help to share more details about the problem.
>From the sources jars I noticed that 'send()' method uses 'sendAsync()' method so can you please confirm with the 'send()' method that are we using sync mode or async mode
I can confirm your observation that send uses sendAsync in the implementation. When you are using "send", you are using the synchronous API call and when you are calling "sendAsync", it's an asynchronous API call, that returns a CompletableFuture. I hope this helps.
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8994562
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
GitHub user lhotari added a comment to the discussion: async/sync mode in pulsar producer and consumer
When sending messages asynchronously, you cannot assume that they are sent from the client to the broker until the callback receives a message id from each message that has been sent. This comment about the correct usage of the Pulsar Java client might be helpful: https://github.com/apache/pulsar/pull/22011#issuecomment-1936188709
The relevant part:
> The Pulsar client is designed to continue attempting to send messages until a potential send timeout occurs. It's also possible to set up an unlimited send timeout, allowing the client to retry indefinitely. This feature is detailed in the Pulsar documentation, available at https://pulsar.apache.org/docs/3.1.x/cookbooks-deduplication/#pulsar-clients (it's explained in the context of message deduplication). You can refer to the Javadocs for [sendTimeout on ProducerBuilder](https://pulsar.apache.org/api/client/3.1.x/org/apache/pulsar/client/api/ProducerBuilder.html#sendTimeout(int,java.util.concurrent.TimeUnit)).
> It's crucial for messaging applications to be equipped to handle potential failures in message delivery, especially when data consistency is a key concern. Once the Pulsar client has acknowledged the message as sent by returning the message id, the responsibility for maintaining and ensuring the delivery of the message shifts to Pulsar. It's also necessary to verify that the message id is returned when using the asynchronous API ([sendAsync](https://pulsar.apache.org/api/client/3.1.x/org/apache/pulsar/client/api/Producer.html#sendAsync(T))).
> If sending results in an error or the messaging application never receives a message id from the Pulsar client, it's the messaging application's responsibility to retry.
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8981176
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "ragaur-tibco (via GitHub)" <gi...@apache.org>.
GitHub user ragaur-tibco added a comment to the discussion: async/sync mode in pulsar producer and consumer
@lhotari How can I determine if a Pulsar producer supports both synchronous and asynchronous modes, especially when the `send()` method internally calls the `sendAsync()` method?
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8981497
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
GitHub user lhotari added a comment to the discussion: async/sync mode in pulsar producer and consumer
@ragaur-tibco For understanding the "internal process", it is good to start by reading [the high level documentation for the Pulsar binary protocol](https://pulsar.apache.org/docs/3.2.x/developing-binary-protocol/). For example, the [description of the producer](https://pulsar.apache.org/docs/3.2.x/developing-binary-protocol/#producer) gives good context to understand how the producer works. There's a presentation ["Deep Dive into the Pulsar Binary Protocol"](https://www.youtube.com/watch?v=ONyld7Da-rw) by @cbornet which explains a lot more around the binary protocol.
Pulsar's Java client and Pulsar broker use Netty to handle the network communications. If you really want to get into internal, it helps to get an understanding of Netty. One of the rare books on the topic is "[Netty in Action](https://www.manning.com/books/netty-in-action)".
> I don't have any expectations or requirements I just want to know how to test the both modes any practical approach and anything in Pulsar SDK logs so that I can understand the internal process
As I have already mentioned in my previous comments, testing Pulsar is usually about validating some expectation, for example validating that a system meets both functional and non-functional requirements. There might be a test plan with test cases to make this well defined. Perhaps you are meaning more about experimentation when you say to "test the both modes"?
One possible approach is to look at the unit tests in the Pulsar source code and extract something out of that? For example, [this test class contains a lot of different test cases](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java).
With ChatGPT, you can also generate a lot of examples and just try to run them in the IDE with unit tests, if your main purpose is to experiment with things and learn Pulsar.
For Pulsar testing in general, you might get good ideas [from some conference talks](https://www.youtube.com/@streamnative/search?query=testing). I've done a talk ["Validating Apache Pulsar’s Behavior Under Failure Conditions "](https://www.youtube.com/watch?v=98wT5ePSKj0) on this subject, but it's a fairly high-level systems testing talk so it might not be practical for your case.
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8993754
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "visortelle (via GitHub)" <gi...@apache.org>.
GitHub user visortelle edited a comment on the discussion: async/sync mode in pulsar producer and consumer
@ragaur-tibco I read the whole discussion 3 times and am still unsure I understood your question.
> I don't have any expectations or requirements I just want to know **how to test the both modes** any practical approach and anything in Pulsar SDK logs so that I can understand the internal process
@lhotari provided the file full of related test cases: https://github.com/apache/pulsar/blob/master/pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java
I'm sure you'll find the answer to your question here.
Closing for now.
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8994539
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "ragaur-tibco (via GitHub)" <gi...@apache.org>.
GitHub user ragaur-tibco closed a discussion: async/sync mode in pulsar producer and consumer
What is the best way to check and confirm that Async/Sync mode is working in Pulsar producer and consumer
GitHub link: https://github.com/apache/pulsar/discussions/22382
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
GitHub user lhotari added a comment to the discussion: async/sync mode in pulsar producer and consumer
One more clarification needed: please explain what "working" means to you when you say "does async and sync mode work and how can I verify that the mode selected is working"?
Do you want to understand how to ensure that a message is successfully accepted for delivery by the broker?
Perhaps it would help to describe your use case. What are you trying to achieve?
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8980760
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] async/sync mode in pulsar producer and consumer [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
GitHub user lhotari edited a comment on the discussion: async/sync mode in pulsar producer and consumer
When sending messages asynchronously, you cannot assume that they are sent from the client to the broker until the callback receives a message id from each message that has been sent. This comment about the correct usage of the Pulsar Java client might be helpful: https://github.com/apache/pulsar/pull/22011#issuecomment-1936188709
The relevant part:
> The Pulsar client is designed to continue attempting to send messages until a potential send timeout occurs. It's also possible to set up an unlimited send timeout, allowing the client to retry indefinitely. This feature is detailed in the Pulsar documentation, available at https://pulsar.apache.org/docs/3.1.x/cookbooks-deduplication/#pulsar-clients (it's explained in the context of message deduplication). You can refer to the Javadocs for [sendTimeout on ProducerBuilder](https://pulsar.apache.org/api/client/3.1.x/org/apache/pulsar/client/api/ProducerBuilder.html#sendTimeout(int,java.util.concurrent.TimeUnit)).
>
> It's crucial for messaging applications to be equipped to handle potential failures in message delivery, especially when data consistency is a key concern. Once the Pulsar client has acknowledged the message as sent by returning the message id, the responsibility for maintaining and ensuring the delivery of the message shifts to Pulsar. It's also necessary to verify that the message id is returned when using the asynchronous API ([sendAsync](https://pulsar.apache.org/api/client/3.1.x/org/apache/pulsar/client/api/Producer.html#sendAsync(T))).
>
> If sending results in an error or the messaging application never receives a message id from the Pulsar client, it's the messaging application's responsibility to retry.
GitHub link: https://github.com/apache/pulsar/discussions/22382#discussioncomment-8981176
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org