You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by Hang Chen <ch...@apache.org> on 2023/03/20 13:05:28 UTC
[VOTE] BP-62: Streamline batch add requests
This is the vote for BP-62.
### Motivation
The bookie server process add-entry requests pipeline:
- Get one request from the Netty socket channel
- Choose one thread to process the written request
- Write the entry into the target ledger entry logger's write
cache(memory table)
- Put the entry into the journal's pending queue
- Journal thread takes the entry from the pending queue and writes it
into PageCache/Journal Disk
- Write the callback response to the Netty buffer and flush to the
bookie client side.
For every add entry request, the bookie server needs to go through the
above steps one by one. It will introduce a lot of thread context
switches.
We can batch the add requests according to the Netty socket channel,
and write a batch of entries into the ledger entry logger and journal
disk.
### Modifications
The PR will change the add requests processing pipeline into the
following steps.
- Get a batch of add-entry requests from the socket channel until the
socket channel is empty or reached the max capacity (default is 1_000)
- Choose one thread to process the batch of add-entry requests.
- Write the entries into the target ledger entry logger's write cache one by one
- Put the batch of entries into the journal's pending queue
- Journal thread drains a batch of entries from the pending queue and
writes them into PageCache/Journal disk
- Write the callback response to the Netty buffer and flush to the
bookie client side.
With this change, we can save a lot of thread context switches.
### Performance
I start one bookie on my laptop and use the Bookkeeper benchmark to
test the performance
```shell
bin/benchmark writes -ensemble 1 -quorum 1 -ackQuorum 1 -ledgers 50
-throttle 20000
```
**Before this change**
| times | ops/sec | p95 latency | p99 latency |
| --- | --- | --- | --- |
| 1 | 147507 | 114.93 | 122.42 |
| 2 | 154571 | 111.46 | 115.86 |
| 3 | 141459 | 117.23 | 124.18 |
| 4 | 142037 | 121.75 | 128.54 |
| 5 | 143682 | 121.05 | 127.97 |
**After this change**
| times | ops/sec | p95 latency | p99 latency |
| --- | --- | --- | --- |
| 1 | 157328 | 118.30 | 121.79 |
| 2 | 165774 | 112.86 | 115.69 |
| 3 | 144790 | 128.94 | 133.24 |
| 4 | 151984 | 121.88 | 125.32 |
| 5 | 154574 | 121.57 | 124.57 |
The new change has a 2.2% improvement.
Please leave +1/-1 in this thread to join the vote. And feel free to
leave any concerns.
Thanks,
Hang
Re: [VOTE] BP-62: Streamline batch add requests
Posted by Wenbing Shen <ol...@gmail.com>.
+1(non-binding)
Thanks
Wenbing
ZhangJian He <sh...@gmail.com> 于2023年3月21日周二 09:53写道：
> +1(non-binding)
>
> Thanks
> ZhangJian He
>
>
> On Tue, 21 Mar 2023 at 08:11, Andrey Yegorov <an...@datastax.com>
> wrote:
>
> > Overall, +1.
> > I do have some concerns related to handling various limiters, so I left a
> > comment on the PR.
> >
> > On Mon, Mar 20, 2023 at 6:05 AM Hang Chen <ch...@apache.org> wrote:
> >
> > > This is the vote for BP-62.
> > >
> > > ### Motivation
> > > The bookie server process add-entry requests pipeline:
> > > - Get one request from the Netty socket channel
> > > - Choose one thread to process the written request
> > > - Write the entry into the target ledger entry logger's write
> > > cache(memory table)
> > > - Put the entry into the journal's pending queue
> > > - Journal thread takes the entry from the pending queue and writes it
> > > into PageCache/Journal Disk
> > > - Write the callback response to the Netty buffer and flush to the
> > > bookie client side.
> > >
> > > For every add entry request, the bookie server needs to go through the
> > > above steps one by one. It will introduce a lot of thread context
> > > switches.
> > >
> > > We can batch the add requests according to the Netty socket channel,
> > > and write a batch of entries into the ledger entry logger and journal
> > > disk.
> > >
> > > ### Modifications
> > > The PR will change the add requests processing pipeline into the
> > > following steps.
> > > - Get a batch of add-entry requests from the socket channel until the
> > > socket channel is empty or reached the max capacity (default is 1_000)
> > > - Choose one thread to process the batch of add-entry requests.
> > > - Write the entries into the target ledger entry logger's write cache
> one
> > > by one
> > > - Put the batch of entries into the journal's pending queue
> > > - Journal thread drains a batch of entries from the pending queue and
> > > writes them into PageCache/Journal disk
> > > - Write the callback response to the Netty buffer and flush to the
> > > bookie client side.
> > >
> > > With this change, we can save a lot of thread context switches.
> > >
> > >
> > > ### Performance
> > > I start one bookie on my laptop and use the Bookkeeper benchmark to
> > > test the performance
> > > ```shell
> > > bin/benchmark writes -ensemble 1 -quorum 1 -ackQuorum 1 -ledgers 50
> > > -throttle 20000
> > > ```
> > >
> > > **Before this change**
> > >
> > > | times | ops/sec | p95 latency | p99 latency |
> > > | --- | --- | --- | --- |
> > > | 1 | 147507 | 114.93 | 122.42 |
> > > | 2 | 154571 | 111.46 | 115.86 |
> > > | 3 | 141459 | 117.23 | 124.18 |
> > > | 4 | 142037 | 121.75 | 128.54 |
> > > | 5 | 143682 | 121.05 | 127.97 |
> > >
> > >
> > > **After this change**
> > >
> > > | times | ops/sec | p95 latency | p99 latency |
> > > | --- | --- | --- | --- |
> > > | 1 | 157328 | 118.30 | 121.79 |
> > > | 2 | 165774 | 112.86 | 115.69 |
> > > | 3 | 144790 | 128.94 | 133.24 |
> > > | 4 | 151984 | 121.88 | 125.32 |
> > > | 5 | 154574 | 121.57 | 124.57 |
> > >
> > > The new change has a 2.2% improvement.
> > >
> > > Please leave +1/-1 in this thread to join the vote. And feel free to
> > > leave any concerns.
> > >
> > > Thanks,
> > > Hang
> > >
> >
> >
> > --
> > Andrey Yegorov
> >
>
Re: [VOTE] BP-62: Streamline batch add requests
Posted by ZhangJian He <sh...@gmail.com>.
+1(non-binding)
Thanks
ZhangJian He
On Tue, 21 Mar 2023 at 08:11, Andrey Yegorov <an...@datastax.com>
wrote:
> Overall, +1.
> I do have some concerns related to handling various limiters, so I left a
> comment on the PR.
>
> On Mon, Mar 20, 2023 at 6:05 AM Hang Chen <ch...@apache.org> wrote:
>
> > This is the vote for BP-62.
> >
> > ### Motivation
> > The bookie server process add-entry requests pipeline:
> > - Get one request from the Netty socket channel
> > - Choose one thread to process the written request
> > - Write the entry into the target ledger entry logger's write
> > cache(memory table)
> > - Put the entry into the journal's pending queue
> > - Journal thread takes the entry from the pending queue and writes it
> > into PageCache/Journal Disk
> > - Write the callback response to the Netty buffer and flush to the
> > bookie client side.
> >
> > For every add entry request, the bookie server needs to go through the
> > above steps one by one. It will introduce a lot of thread context
> > switches.
> >
> > We can batch the add requests according to the Netty socket channel,
> > and write a batch of entries into the ledger entry logger and journal
> > disk.
> >
> > ### Modifications
> > The PR will change the add requests processing pipeline into the
> > following steps.
> > - Get a batch of add-entry requests from the socket channel until the
> > socket channel is empty or reached the max capacity (default is 1_000)
> > - Choose one thread to process the batch of add-entry requests.
> > - Write the entries into the target ledger entry logger's write cache one
> > by one
> > - Put the batch of entries into the journal's pending queue
> > - Journal thread drains a batch of entries from the pending queue and
> > writes them into PageCache/Journal disk
> > - Write the callback response to the Netty buffer and flush to the
> > bookie client side.
> >
> > With this change, we can save a lot of thread context switches.
> >
> >
> > ### Performance
> > I start one bookie on my laptop and use the Bookkeeper benchmark to
> > test the performance
> > ```shell
> > bin/benchmark writes -ensemble 1 -quorum 1 -ackQuorum 1 -ledgers 50
> > -throttle 20000
> > ```
> >
> > **Before this change**
> >
> > | times | ops/sec | p95 latency | p99 latency |
> > | --- | --- | --- | --- |
> > | 1 | 147507 | 114.93 | 122.42 |
> > | 2 | 154571 | 111.46 | 115.86 |
> > | 3 | 141459 | 117.23 | 124.18 |
> > | 4 | 142037 | 121.75 | 128.54 |
> > | 5 | 143682 | 121.05 | 127.97 |
> >
> >
> > **After this change**
> >
> > | times | ops/sec | p95 latency | p99 latency |
> > | --- | --- | --- | --- |
> > | 1 | 157328 | 118.30 | 121.79 |
> > | 2 | 165774 | 112.86 | 115.69 |
> > | 3 | 144790 | 128.94 | 133.24 |
> > | 4 | 151984 | 121.88 | 125.32 |
> > | 5 | 154574 | 121.57 | 124.57 |
> >
> > The new change has a 2.2% improvement.
> >
> > Please leave +1/-1 in this thread to join the vote. And feel free to
> > leave any concerns.
> >
> > Thanks,
> > Hang
> >
>
>
> --
> Andrey Yegorov
>
Re: [VOTE] BP-62: Streamline batch add requests
Posted by Andrey Yegorov <an...@datastax.com>.
Overall, +1.
I do have some concerns related to handling various limiters, so I left a
comment on the PR.
On Mon, Mar 20, 2023 at 6:05 AM Hang Chen <ch...@apache.org> wrote:
> This is the vote for BP-62.
>
> ### Motivation
> The bookie server process add-entry requests pipeline:
> - Get one request from the Netty socket channel
> - Choose one thread to process the written request
> - Write the entry into the target ledger entry logger's write
> cache(memory table)
> - Put the entry into the journal's pending queue
> - Journal thread takes the entry from the pending queue and writes it
> into PageCache/Journal Disk
> - Write the callback response to the Netty buffer and flush to the
> bookie client side.
>
> For every add entry request, the bookie server needs to go through the
> above steps one by one. It will introduce a lot of thread context
> switches.
>
> We can batch the add requests according to the Netty socket channel,
> and write a batch of entries into the ledger entry logger and journal
> disk.
>
> ### Modifications
> The PR will change the add requests processing pipeline into the
> following steps.
> - Get a batch of add-entry requests from the socket channel until the
> socket channel is empty or reached the max capacity (default is 1_000)
> - Choose one thread to process the batch of add-entry requests.
> - Write the entries into the target ledger entry logger's write cache one
> by one
> - Put the batch of entries into the journal's pending queue
> - Journal thread drains a batch of entries from the pending queue and
> writes them into PageCache/Journal disk
> - Write the callback response to the Netty buffer and flush to the
> bookie client side.
>
> With this change, we can save a lot of thread context switches.
>
>
> ### Performance
> I start one bookie on my laptop and use the Bookkeeper benchmark to
> test the performance
> ```shell
> bin/benchmark writes -ensemble 1 -quorum 1 -ackQuorum 1 -ledgers 50
> -throttle 20000
> ```
>
> **Before this change**
>
> | times | ops/sec | p95 latency | p99 latency |
> | --- | --- | --- | --- |
> | 1 | 147507 | 114.93 | 122.42 |
> | 2 | 154571 | 111.46 | 115.86 |
> | 3 | 141459 | 117.23 | 124.18 |
> | 4 | 142037 | 121.75 | 128.54 |
> | 5 | 143682 | 121.05 | 127.97 |
>
>
> **After this change**
>
> | times | ops/sec | p95 latency | p99 latency |
> | --- | --- | --- | --- |
> | 1 | 157328 | 118.30 | 121.79 |
> | 2 | 165774 | 112.86 | 115.69 |
> | 3 | 144790 | 128.94 | 133.24 |
> | 4 | 151984 | 121.88 | 125.32 |
> | 5 | 154574 | 121.57 | 124.57 |
>
> The new change has a 2.2% improvement.
>
> Please leave +1/-1 in this thread to join the vote. And feel free to
> leave any concerns.
>
> Thanks,
> Hang
>
--
Andrey Yegorov
Re: [VOTE] BP-62: Streamline batch add requests
Posted by Matteo Merli <ma...@gmail.com>.
+1
--
Matteo Merli
<ma...@gmail.com>
On Mon, Mar 20, 2023 at 6:05 AM Hang Chen <ch...@apache.org> wrote:
> This is the vote for BP-62.
>
> ### Motivation
> The bookie server process add-entry requests pipeline:
> - Get one request from the Netty socket channel
> - Choose one thread to process the written request
> - Write the entry into the target ledger entry logger's write
> cache(memory table)
> - Put the entry into the journal's pending queue
> - Journal thread takes the entry from the pending queue and writes it
> into PageCache/Journal Disk
> - Write the callback response to the Netty buffer and flush to the
> bookie client side.
>
> For every add entry request, the bookie server needs to go through the
> above steps one by one. It will introduce a lot of thread context
> switches.
>
> We can batch the add requests according to the Netty socket channel,
> and write a batch of entries into the ledger entry logger and journal
> disk.
>
> ### Modifications
> The PR will change the add requests processing pipeline into the
> following steps.
> - Get a batch of add-entry requests from the socket channel until the
> socket channel is empty or reached the max capacity (default is 1_000)
> - Choose one thread to process the batch of add-entry requests.
> - Write the entries into the target ledger entry logger's write cache one
> by one
> - Put the batch of entries into the journal's pending queue
> - Journal thread drains a batch of entries from the pending queue and
> writes them into PageCache/Journal disk
> - Write the callback response to the Netty buffer and flush to the
> bookie client side.
>
> With this change, we can save a lot of thread context switches.
>
>
> ### Performance
> I start one bookie on my laptop and use the Bookkeeper benchmark to
> test the performance
> ```shell
> bin/benchmark writes -ensemble 1 -quorum 1 -ackQuorum 1 -ledgers 50
> -throttle 20000
> ```
>
> **Before this change**
>
> | times | ops/sec | p95 latency | p99 latency |
> | --- | --- | --- | --- |
> | 1 | 147507 | 114.93 | 122.42 |
> | 2 | 154571 | 111.46 | 115.86 |
> | 3 | 141459 | 117.23 | 124.18 |
> | 4 | 142037 | 121.75 | 128.54 |
> | 5 | 143682 | 121.05 | 127.97 |
>
>
> **After this change**
>
> | times | ops/sec | p95 latency | p99 latency |
> | --- | --- | --- | --- |
> | 1 | 157328 | 118.30 | 121.79 |
> | 2 | 165774 | 112.86 | 115.69 |
> | 3 | 144790 | 128.94 | 133.24 |
> | 4 | 151984 | 121.88 | 125.32 |
> | 5 | 154574 | 121.57 | 124.57 |
>
> The new change has a 2.2% improvement.
>
> Please leave +1/-1 in this thread to join the vote. And feel free to
> leave any concerns.
>
> Thanks,
> Hang
>