[DISCUSS] Back pressure in Apache Pulsar

[mattison chao <ma...@apache.org>]

Hi, All

Since Pulsar has many asynchronous operations, we need to talk about
backpressure. Because this is something to watch out for in async and I've
seen comments about it in some other PRs.

I think we need to open a separate discussion to collect the problems we
are having and the solutions we need to provide.


Best.
Mattison

FYI:
https://github.com/apache/pulsar/issues/16998#issuecomment-1244833965
https://github.com/apache/pulsar/pull/17349#discussion_r964399426

Re: [DISCUSS] Back pressure in Apache Pulsar

[Asaf Mesika <as...@gmail.com>]

Mattison, can you provide more context here?

1st link describes a complicated situation in which the BK node Pulsar is
working against is overwhelmed, and we would like to transmit this
back-pressure back to consumers/producers. This creates an issue as you
need to notify the load balancer of this, as this might get resolved if the
topic moves to another broker, perhaps?

2nd link talks about submitting many tasks to an executor service and
having no means to control the number of in-flight requests (which I
thought the executor service's capacity gives you exactly that.



On 13 Sep 2022 at 7:35:46, mattison chao <ma...@apache.org> wrote:

> Hi, All
>
> Since Pulsar has many asynchronous operations, we need to talk about
> backpressure. Because this is something to watch out for in async and I've
> seen comments about it in some other PRs.
>
> I think we need to open a separate discussion to collect the problems we
> are having and the solutions we need to provide.
>
>
> Best.
> Mattison
>
> FYI:
> https://github.com/apache/pulsar/issues/16998#issuecomment-1244833965
> https://github.com/apache/pulsar/pull/17349#discussion_r964399426
>

Re: [DISCUSS] Back pressure in Apache Pulsar

[Lari Hotari <lh...@apache.org>]

After my previous email, I received a question about Pulsar
backpressure solutions:
"Why can't we use Reactive frameworks here?"

That's a good question.

Pulsar isn't implemented in a way where Reactive frameworks would be
easily applicable or make sense in the Broker. (For implementing the
REST API, that would be a different story.) There could also be
performance reasons which make some Reactive frameworks not a good fit
for implementing low-level network servers.

Apple ServiceTalk is a framework which does support Reactive Streams and
could be a reasonable solution for implementing network servers. quoting
https://opensource.apple.com/projects/servicetalk/:

"ServiceTalk is built by the team that leads the Netty project and is
intended to be the next layer in the stack. ServiceTalk's asynchronous
primitives follow the Reactive Streams specification and propagate
backpressure/cancellation for your application. Asynchronous control
flow can be challenging and not necessary for all use cases. For this
reason ServiceTalk provides blocking APIs and allows you to choose
if/when to evolve to asynchronous per-route in a safe manner."

Reactive Streams provides a solution for non-blocking asynchronous
backpressure. A plain CompletableFuture based programming model doesn't
have that. Apple Servicetalk docs related to Reactive:
https://apple.github.io/servicetalk//servicetalk-concurrent-api/SNAPSHOT/asynchronous-primitives.html

This is a sidetrack. I think it would be more practical to focus on
describing the current backpressure solution in Pulsar Broker than
thinking about possible future solutions.

-Lari

On Wed, Sep 14, 2022 at 2:59 PM Lari Hotari <lh...@apache.org> wrote:
>
> Thank you Mattison for starting the discussion.
>
> Just adding some more context.
>
> In general, every type of processing pipeline solution needs a solution
> for backpressure. This could mean different types of solutions for
> different goals, depending on the context. Usually a backpressure
> solution is some type of flow control solution (wikipedia:
> https://en.wikipedia.org/wiki/Flow_control_(data) ).
> There are multiple backpressure solutions in Pulsar.
>
> For the broker, flow control is typically achieved using throttling
> based on explicit rate limits and there's also the solution of
> feedback-based flow control in the Pulsar binary protocol between the
> broker and consumers, where "permits" are exchanged between the broker
> and the consumer (document:
> https://pulsar.apache.org/docs/develop-binary-protocol/#flow-control )
>
> In Netty servers and clients, a common technical solution to
> backpressure is to toggle the Netty "auto read" state of the TCP/IP
> connection to pause or resume reading more input. It is also common in
> Netty servers and clients to stop reading input from the TCP/IP
> connection when the output buffer of the connection is full and resume
> when the output buffer level goes below a threshold.
>
> Pulsar uses TCP/IP connection level controls for achieving backpressure
> for producers. The challenge is that Pulsar can share a single TCP/IP
> connection across multiple producers and consumers. Because of this,
> there could be multiple producer and dispatcher rate limiters operating
> on the same connection on the broker, and they will do conflicting
> decisions, which results in inconsistent behavior.
>
> For backpressure between Pulsar and Bookkeepers, there are multiple
> backpressure/flow control solutions. There's also an explicit
> backpressure feature in Bookkeeper that could be enabled on server and
> client side. There is an issue
> https://github.com/apache/pulsar/issues/10439 to track enabling this
> feature in Pulsar Broker and in the default configuration for Bookkeeper
> as part of Pulsar deployments.
>
> Even without this explicit feature, there are forms of backpressure in
> place between brokers and bookies.
>
> It would be helpful if the flow control between brokers and bookies
> would be explained and documented in detail. This would help both Pulsar
> users and developers to build understanding about the performance model
> of Pulsar.
>
> One properly of flow control is that it can fail if the whole pipeline
> isn't covered from end-to-end. Gaps in flow control could also lead to
> amplification issues where the system has to cope with even more load
> because retries that clients send when previous requests timeout or
> fail. There could also be thundering herds issues where the system gets
> high spikes of traffic when requests get buffered because of various
> reasons and this could lead to cascading failures when the total load of
> the system increases in unexpected ways because of amplification
> effects.
>
> Touching upon Pulsar Admin REST API. The context for back pressure is a
> lot different in the Pulsar Admin REST API. Before the PIP-149 async
> changes, there was explicit backpressure in the REST API implementation.
> The system could handle a limited amount of work and it would process
> downstream work items one-by-one.
>
> With "PIP-149 Making the REST Admin API fully async"
> (https://github.com/apache/pulsar/issues/14365), there are different
> challenges related to backpressure. It is usually about how to limit the
> in-progress work in the system. An async system will accept a lot of
> work compared to the previous solution and this accepted work will get
> processed in the async REST API backend eventually even when the clients
> have already closed the connection and sent a new retry. One possible
> solution to this issue is to limit incoming requests at the HTTP server
> level with features that Jetty provides for limiting concurrency. PRs
> https://github.com/apache/pulsar/pull/14353 and
> https://github.com/apache/pulsar/pull/15637 added this support to
> Pulsar. The values might have to be tuned to a lot lower values to
> prevent issues in practice. This is not a complete solution for REST API
> backend. It would be useful to also have a solution that would cancel
> down stream requests that are for incoming HTTP requests that no longer
> exist since the client stopped waiting for the response. The main down
> stream requests are towards the metadata store. It might also be
> necessary to limit the number of outstanding downstream requests. With
> batching in metadata store, that might not be an issue.
>
> I hope it was useful to share some thoughts around backpressure in
> Pulsar. I'm looking forward to learning more from others about the
> design for backpressure in the Pulsar broker. Please share your thoughts
> too!
>
> -Lari
>
>
>
>
>
> On Tue, Sep 13, 2022 at 7:36 AM mattison chao <ma...@apache.org> wrote:
> >
> > Hi, All
> >
> > Since Pulsar has many asynchronous operations, we need to talk about
> > backpressure. Because this is something to watch out for in async and I've
> > seen comments about it in some other PRs.
> >
> > I think we need to open a separate discussion to collect the problems we
> > are having and the solutions we need to provide.
> >
> >
> > Best.
> > Mattison
> >
> > FYI:
> > https://github.com/apache/pulsar/issues/16998#issuecomment-1244833965
> > https://github.com/apache/pulsar/pull/17349#discussion_r964399426

Re: [DISCUSS] Back pressure in Apache Pulsar

[Lari Hotari <lh...@apache.org>]

Thank you Mattison for starting the discussion.

Just adding some more context.

In general, every type of processing pipeline solution needs a solution
for backpressure. This could mean different types of solutions for
different goals, depending on the context. Usually a backpressure
solution is some type of flow control solution (wikipedia:
https://en.wikipedia.org/wiki/Flow_control_(data) ).
There are multiple backpressure solutions in Pulsar.

For the broker, flow control is typically achieved using throttling
based on explicit rate limits and there's also the solution of
feedback-based flow control in the Pulsar binary protocol between the
broker and consumers, where "permits" are exchanged between the broker
and the consumer (document:
https://pulsar.apache.org/docs/develop-binary-protocol/#flow-control )

In Netty servers and clients, a common technical solution to
backpressure is to toggle the Netty "auto read" state of the TCP/IP
connection to pause or resume reading more input. It is also common in
Netty servers and clients to stop reading input from the TCP/IP
connection when the output buffer of the connection is full and resume
when the output buffer level goes below a threshold.

Pulsar uses TCP/IP connection level controls for achieving backpressure
for producers. The challenge is that Pulsar can share a single TCP/IP
connection across multiple producers and consumers. Because of this,
there could be multiple producer and dispatcher rate limiters operating
on the same connection on the broker, and they will do conflicting
decisions, which results in inconsistent behavior.

For backpressure between Pulsar and Bookkeepers, there are multiple
backpressure/flow control solutions. There's also an explicit
backpressure feature in Bookkeeper that could be enabled on server and
client side. There is an issue
https://github.com/apache/pulsar/issues/10439 to track enabling this
feature in Pulsar Broker and in the default configuration for Bookkeeper
as part of Pulsar deployments.

Even without this explicit feature, there are forms of backpressure in
place between brokers and bookies.

It would be helpful if the flow control between brokers and bookies
would be explained and documented in detail. This would help both Pulsar
users and developers to build understanding about the performance model
of Pulsar.

One properly of flow control is that it can fail if the whole pipeline
isn't covered from end-to-end. Gaps in flow control could also lead to
amplification issues where the system has to cope with even more load
because retries that clients send when previous requests timeout or
fail. There could also be thundering herds issues where the system gets
high spikes of traffic when requests get buffered because of various
reasons and this could lead to cascading failures when the total load of
the system increases in unexpected ways because of amplification
effects.

Touching upon Pulsar Admin REST API. The context for back pressure is a
lot different in the Pulsar Admin REST API. Before the PIP-149 async
changes, there was explicit backpressure in the REST API implementation.
The system could handle a limited amount of work and it would process
downstream work items one-by-one.

With "PIP-149 Making the REST Admin API fully async"
(https://github.com/apache/pulsar/issues/14365), there are different
challenges related to backpressure. It is usually about how to limit the
in-progress work in the system. An async system will accept a lot of
work compared to the previous solution and this accepted work will get
processed in the async REST API backend eventually even when the clients
have already closed the connection and sent a new retry. One possible
solution to this issue is to limit incoming requests at the HTTP server
level with features that Jetty provides for limiting concurrency. PRs
https://github.com/apache/pulsar/pull/14353 and
https://github.com/apache/pulsar/pull/15637 added this support to
Pulsar. The values might have to be tuned to a lot lower values to
prevent issues in practice. This is not a complete solution for REST API
backend. It would be useful to also have a solution that would cancel
down stream requests that are for incoming HTTP requests that no longer
exist since the client stopped waiting for the response. The main down
stream requests are towards the metadata store. It might also be
necessary to limit the number of outstanding downstream requests. With
batching in metadata store, that might not be an issue.

I hope it was useful to share some thoughts around backpressure in
Pulsar. I'm looking forward to learning more from others about the
design for backpressure in the Pulsar broker. Please share your thoughts
too!

-Lari





On Tue, Sep 13, 2022 at 7:36 AM mattison chao <ma...@apache.org> wrote:
>
> Hi, All
>
> Since Pulsar has many asynchronous operations, we need to talk about
> backpressure. Because this is something to watch out for in async and I've
> seen comments about it in some other PRs.
>
> I think we need to open a separate discussion to collect the problems we
> are having and the solutions we need to provide.
>
>
> Best.
> Mattison
>
> FYI:
> https://github.com/apache/pulsar/issues/16998#issuecomment-1244833965
> https://github.com/apache/pulsar/pull/17349#discussion_r964399426