Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

[Matteo Merli <ma...@gmail.com>]

--
Matteo Merli
<ma...@gmail.com>


On Tue, Aug 8, 2023 at 9:56 PM Michael Marshall <mm...@apache.org>
wrote:

> Thanks for your proposal, Heesung. Sorry for my late response. Just
> want to make a few points to hopefully improve the implementation.
> Overall, I think this feature is absolutely the right direction for
> Pulsar and will be a great improvement. I remember discussing this at
> some community meetings last year. See the last paragraph of [0].
>
> In general, my primary question is about the protocol changes you're
> proposing, both the commands and the way the commands are sent.
>
> > I am worried about the backward compatibility if we add a new proto
> message
> > -- how can the old versions of clients handle this new proto message?
>
> The protocolVersion [1] object in the proto definition is how we
> guarantee backwards and forwards compatibility for all combinations of
> brokers/clients. The client and the broker each share their version
> during the handshake and then they only use commands known to both
> peers.


> We can introduce a new command here if we want/need. The nuance is
> that the server logic will branch depending on the client's
> protocolVersion.
>

> I don't see versioning the protocol as a negative as long as there is
> sufficient benefit. In this case, the new command serves an explicit
> purpose to make the protocol more reactive (the client knows what is
> happening as it happens without any need to poll) and less wasteful
> (no unnecessary data transfer).
>
> One reason I want to push on this design is because I think it will be
> harder to change once it is in place. Adding the new broker URL to the
> close producer and close consumer commands necessarily couples the
> events. Using separate commands allows them to be decoupled, and in my
> opinion, that matches the reality of the problem. We know we want to
> close the producer/consumer before we know where they should connect
> next.


Indeed adding a new command is a possibility, and the protocol has the
option to have feature flags so that each side can understand whether a
feature is supported by the other side.

My view here is that having a mechanism to notify the producers to pause
sending messages has only a very little benefit and multiple drawbacks.

Benefit: if we tell the producer to pause sending (while the topic is
moved), we would spare some network traffic (producer -> broker)

Drawbacks:
 1. Brokers need to be able to handle both cases
     a. Send a pause-producer command and ignore any in-inflight messages
     b. If the producer does not support this feature, the broker would
have to ignore messages anyway
  2. This change would create multiple complications in the clients
implementations:
     a. A new state to take care of
     b. What to do with the pending messages, eg: manage and addition
pending messages queue
  3. What to do with consumers (eg: broker would still have to ignore the
acks while the topic is closing)

I don't see the sparing of network traffic during the failover as a huge
benefit that justifies the added complexity in broker & client libraries.
In both cases, once the topic is recovered in the new broker, all the same
pending messages will have to be resent to the broker and the same
bandwidth will be used there.

All in all, I don't see the close-producer and redirect as two separate
operations.The close is piggy-backing the information to skip the service
discovery. If the clients supports it, it will make use of it, otherwise
fall back to current behavior and do a lookup.

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

[Michael Marshall <mm...@apache.org>]

Thanks for your response, Matteo.

The benefits of my proposed alternative are dependent on the amount of
time between when a broker stops accepting messages and when the
client learns of the new host for the topic. The main areas this will
matter are failure scenarios where that delta or that "mean time to
recovery" is large. My preference is that we build a resilient and
reactive protocol that does not make assumptions about how quickly
certain states will last, like the length of time it will take for
topic transfer. I am partial to the idea that we should "let it fail".
Part of that is sending events to the relevant parties as we discover
them (like telling producers and consumers to pause) as a way to allow
for graceful handling of events.

> Drawbacks:
> 1. Brokers need to be able to handle both cases
>     a. Send a pause-producer command and ignore any in-inflight messages
>     b. If the producer does not support this feature, the broker would
> have to ignore messages anyway

In my version, the broker would either tell the client to close the
producer or the consumer--just like we do today--or would tell the
producer/consumer to pause. Branching logic in the broker based on the
protocol version doesn't seem complicated to me. We already do this.

>   2. This change would create multiple complications in the clients
> implementations:
>     a. A new state to take care of
>     b. What to do with the pending messages, eg: manage and addition
> pending messages queue

I disagree with point b: clients already have to handle pending
messages while they are disconnected. I agree we'd need a new state,
but it would be a special kind of "reconnecting" state which we
already have for every client.

>   3. What to do with consumers (eg: broker would still have to ignore the
> acks while the topic is closing)

Consumers would behave the same way they do today when they are
disconnected from a broker. Further, consumers could potentially use
the "pause" state to queue their acks to decrease the number of dupes.
By letting the consumer send acks that are explicitly ignored, we are
removing information that clients have access to today. Of course
we're working with at least once messaging, so it's technically
acceptable to let them send acks that will be ignored, but our goal is
to decrease dupes.

> I don't see the sparing of network traffic during the failover as a huge
> benefit that justifies the added complexity in broker & client libraries.

The difference would likely be small on an individual basis, but
larger clusters could really benefit from a nuanced protocol. Given
that the majority of messaging system costs come from network transfer
across AZ boundaries, I would think we should prioritize efficiency
over simplicity of design when the cost is only an extra state and
protocol message.

Ultimately, I don't see support for my proposed alternative, so I am
not going to push for my idea further unless prompted. Perhaps one
request might be to add a metric to track how long we "ignore protocol
messages" from producers and consumers so we can know the impact and
that data can influence future design decisions.

Thank you for the discussion, Matteo and Heesung.

- Michael

On Tue, Sep 5, 2023 at 8:08 PM Matteo Merli <ma...@gmail.com> wrote:
>
> --
> Matteo Merli
> <ma...@gmail.com>
>
>
> On Tue, Aug 8, 2023 at 9:56 PM Michael Marshall <mm...@apache.org>
> wrote:
>
> > Thanks for your proposal, Heesung. Sorry for my late response. Just
> > want to make a few points to hopefully improve the implementation.
> > Overall, I think this feature is absolutely the right direction for
> > Pulsar and will be a great improvement. I remember discussing this at
> > some community meetings last year. See the last paragraph of [0].
> >
> > In general, my primary question is about the protocol changes you're
> > proposing, both the commands and the way the commands are sent.
> >
> > > I am worried about the backward compatibility if we add a new proto
> > message
> > > -- how can the old versions of clients handle this new proto message?
> >
> > The protocolVersion [1] object in the proto definition is how we
> > guarantee backwards and forwards compatibility for all combinations of
> > brokers/clients. The client and the broker each share their version
> > during the handshake and then they only use commands known to both
> > peers.
>
>
> > We can introduce a new command here if we want/need. The nuance is
> > that the server logic will branch depending on the client's
> > protocolVersion.
> >
>
> > I don't see versioning the protocol as a negative as long as there is
> > sufficient benefit. In this case, the new command serves an explicit
> > purpose to make the protocol more reactive (the client knows what is
> > happening as it happens without any need to poll) and less wasteful
> > (no unnecessary data transfer).
> >
> > One reason I want to push on this design is because I think it will be
> > harder to change once it is in place. Adding the new broker URL to the
> > close producer and close consumer commands necessarily couples the
> > events. Using separate commands allows them to be decoupled, and in my
> > opinion, that matches the reality of the problem. We know we want to
> > close the producer/consumer before we know where they should connect
> > next.
>
>
> Indeed adding a new command is a possibility, and the protocol has the
> option to have feature flags so that each side can understand whether a
> feature is supported by the other side.
>
> My view here is that having a mechanism to notify the producers to pause
> sending messages has only a very little benefit and multiple drawbacks.
>
> Benefit: if we tell the producer to pause sending (while the topic is
> moved), we would spare some network traffic (producer -> broker)
>
> Drawbacks:
>  1. Brokers need to be able to handle both cases
>      a. Send a pause-producer command and ignore any in-inflight messages
>      b. If the producer does not support this feature, the broker would
> have to ignore messages anyway
>   2. This change would create multiple complications in the clients
> implementations:
>      a. A new state to take care of
>      b. What to do with the pending messages, eg: manage and addition
> pending messages queue
>   3. What to do with consumers (eg: broker would still have to ignore the
> acks while the topic is closing)
>
> I don't see the sparing of network traffic during the failover as a huge
> benefit that justifies the added complexity in broker & client libraries.
> In both cases, once the topic is recovered in the new broker, all the same
> pending messages will have to be resent to the broker and the same
> bandwidth will be used there.
>
> All in all, I don't see the close-producer and redirect as two separate
> operations.The close is piggy-backing the information to skip the service
> discovery. If the clients supports it, it will make use of it, otherwise
> fall back to current behavior and do a lookup.