Pulsar Async Send Mode - Packet Order Guarantees?

[Adam Brenner via users <us...@pulsar.apache.org>]

Hello,

I have a few questions regarding Pulsar Async mode and what, if any, 
guarantees are made with respect to packet ordering:

1) Does Pulsar offer any guarantee packet ordering in Async send mode? 
Let us assume best case with single producer and exclusive access mode.

2) Does the client library do any coordination with Pulsar Broker to 
guarantee packet ordering in async send mode?

3) Would using batch mode with async send mode help in any way to 
guaranteed packet ordering?

Thanks!
-Adam

Re: Pulsar Async Send Mode - Packet Order Guarantees?

[Michael Marshall <mm...@apache.org>]

> Is this because all messages created by the producer are put into a
> message queue for that topic? The work flow being the entire queue is
> flushed to the broker and the client will continue to re-send until
> broker ack's. Is that correct?

Correct.

For the implementation, we rely on TCP to ensure the messages are
ordered while in transit from client to broker (and from broker to
bookkeeper). Then, the client, broker, and bookkeeper are carefully
implemented to ensure order is maintained.

> This is the scenario I was thinking about. Messages 1, 2, 3 are sent and
> ack'ed by the broker. Messages 4, 5, 6 are sent but somehow 4 and 5 did
> not get ack'ed by the broker but 6 did.

This scenario should not happen and would be a bug if it did. Pulsar
is designed so that message 6 can only be persisted if message 4 and
message 5 are persisted.

> With Pulsar and the async message queue, I'm assuming 4, 5 and 6 would
> be sent as one "unit" to the broker so all succeed or do not?

There is no requirement for all to succeed. The only guarantee is that
they will persist in order. As such, 4 could succeed while 5 and 6
fail.

- Michael

On Tue, Mar 14, 2023 at 12:01 PM Adam Brenner via users
<us...@pulsar.apache.org> wrote:
>
> Matteo: thanks for the fast reply! Few follow up questions inline:
>
> On 3/13/23 14:52, Matteo Merli wrote:
> > Hi Adam,
> >
> > On Mon, Mar 13, 2023 at 2:24 PM Adam Brenner via users
> > <users@pulsar.apache.org <ma...@pulsar.apache.org>> wrote:
> >
> >     Hello,
> >
> >     I have a few questions regarding Pulsar Async mode and what, if any,
> >     guarantees are made with respect to packet ordering:
> >
> >     1) Does Pulsar offer any guarantee packet ordering in Async send mode?
> >     Let us assume best case with single producer and exclusive access mode.
> >
> >
> > Yes, messages are guaranteed to be sent in order in both sync & async mode.
>
> Is this because all messages created by the producer are put into a
> message queue for that topic? The work flow being the entire queue is
> flushed to the broker and the client will continue to re-send until
> broker ack's. Is that correct?
>
> Since everything is being operated as a message queue, order is thus
> guaranteed?
>
> >
> >     2) Does the client library do any coordination with Pulsar Broker to
> >     guarantee packet ordering in async send mode?
> >
> >
> > In case of failures, the client library ensures that all messages
> > pending in the producer queue are resent in the same order.
> >
> > This could lead to a sequence of dups, (unless the exactly once mode is
> > enabled), though you wouldn't see a message out of order.
> >
> > NOTE: to have a 100% guarantee you need to disable publish timeout
> > (sendTimeout=0). This is mostly to ease you application code.
> > In the presence of a timeout, Pulsar client guarantees anyway that any
> > message present in the queue will also fail, even if not all of them
> > have already timed out.
>
> > The application though could be already trying to publish the next
> > message. Unless there's proper synchronization. For that, the easier way
> > is to disable timeouts and let the Pulsar client retrying until a
> > successful publish.
>
> This is the scenario I was thinking about. Messages 1, 2, 3 are sent and
> ack'ed by the broker. Messages 4, 5, 6 are sent but somehow 4 and 5 did
> not get ack'ed by the broker but 6 did.
>
> With Pulsar and the async message queue, I'm assuming 4, 5 and 6 would
> be sent as one "unit" to the broker so all succeed or do not?
>
> >
> >
> >     3) Would using batch mode with async send mode help in any way to
> >     guaranteed packet ordering?
> >
> >
> > Batching will improve the throughput and the overall efficiency, less
> > CPU usage, and better compression ratio, though it doesn't affect ordering.
> >
> >
> >
> > --
> > Matteo Merli
> > <mmerli@apache.org <ma...@apache.org>>
>
> Thanks,
> -Adam
>

Re: Pulsar Async Send Mode - Packet Order Guarantees?

[Adam Brenner via users <us...@pulsar.apache.org>]

Matteo: thanks for the fast reply! Few follow up questions inline:

On 3/13/23 14:52, Matteo Merli wrote:
> Hi Adam,
> 
> On Mon, Mar 13, 2023 at 2:24 PM Adam Brenner via users 
> <users@pulsar.apache.org <ma...@pulsar.apache.org>> wrote:
> 
>     Hello,
> 
>     I have a few questions regarding Pulsar Async mode and what, if any,
>     guarantees are made with respect to packet ordering:
> 
>     1) Does Pulsar offer any guarantee packet ordering in Async send mode?
>     Let us assume best case with single producer and exclusive access mode.
> 
> 
> Yes, messages are guaranteed to be sent in order in both sync & async mode.

Is this because all messages created by the producer are put into a 
message queue for that topic? The work flow being the entire queue is 
flushed to the broker and the client will continue to re-send until 
broker ack's. Is that correct?

Since everything is being operated as a message queue, order is thus 
guaranteed?

> 
>     2) Does the client library do any coordination with Pulsar Broker to
>     guarantee packet ordering in async send mode?
> 
> 
> In case of failures, the client library ensures that all messages 
> pending in the producer queue are resent in the same order.
> 
> This could lead to a sequence of dups, (unless the exactly once mode is 
> enabled), though you wouldn't see a message out of order.
> 
> NOTE: to have a 100% guarantee you need to disable publish timeout 
> (sendTimeout=0). This is mostly to ease you application code.
> In the presence of a timeout, Pulsar client guarantees anyway that any 
> message present in the queue will also fail, even if not all of them 
> have already timed out.

> The application though could be already trying to publish the next 
> message. Unless there's proper synchronization. For that, the easier way 
> is to disable timeouts and let the Pulsar client retrying until a 
> successful publish.

This is the scenario I was thinking about. Messages 1, 2, 3 are sent and 
ack'ed by the broker. Messages 4, 5, 6 are sent but somehow 4 and 5 did 
not get ack'ed by the broker but 6 did.

With Pulsar and the async message queue, I'm assuming 4, 5 and 6 would 
be sent as one "unit" to the broker so all succeed or do not?

> 
> 
>     3) Would using batch mode with async send mode help in any way to
>     guaranteed packet ordering?
> 
> 
> Batching will improve the throughput and the overall efficiency, less 
> CPU usage, and better compression ratio, though it doesn't affect ordering.
> 
> 
> 
> --
> Matteo Merli
> <mmerli@apache.org <ma...@apache.org>>

Thanks,
-Adam

Re: Pulsar Async Send Mode - Packet Order Guarantees?

[Matteo Merli <mm...@apache.org>]

Hi Adam,

On Mon, Mar 13, 2023 at 2:24 PM Adam Brenner via users <
users@pulsar.apache.org> wrote:

> Hello,
>
> I have a few questions regarding Pulsar Async mode and what, if any,
> guarantees are made with respect to packet ordering:
>
> 1) Does Pulsar offer any guarantee packet ordering in Async send mode?
> Let us assume best case with single producer and exclusive access mode.
>

Yes, messages are guaranteed to be sent in order in both sync & async mode.


>
> 2) Does the client library do any coordination with Pulsar Broker to
> guarantee packet ordering in async send mode?
>

In case of failures, the client library ensures that all messages pending
in the producer queue are resent in the same order.

This could lead to a sequence of dups, (unless the exactly once mode is
enabled), though you wouldn't see a message out of order.

NOTE: to have a 100% guarantee you need to disable publish timeout
(sendTimeout=0). This is mostly to ease you application code.
In the presence of a timeout, Pulsar client guarantees anyway that any
message present in the queue will also fail, even if not all of them have
already timed out.
The application though could be already trying to publish the next message.
Unless there's proper synchronization. For that, the easier way is to
disable timeouts and let the Pulsar client retrying until a successful
publish.


>
> 3) Would using batch mode with async send mode help in any way to
> guaranteed packet ordering?
>

Batching will improve the throughput and the overall efficiency, less CPU
usage, and better compression ratio, though it doesn't affect ordering.



--
Matteo Merli
<mm...@apache.org>