You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2022/03/07 11:38:43 UTC
[GitHub] [bookkeeper] wuzhanpeng opened a new issue #3085: BP-49: Support reading ahead in async mode
wuzhanpeng opened a new issue #3085:
URL: https://github.com/apache/bookkeeper/issues/3085
**BP**
> Follow the instructions at http://bookkeeper.apache.org/community/bookkeeper_proposals/ to create a proposal.
This is the master ticket for tracking BP-49 :
### Motivation
#### Current Design of Read-ahead
Under the current design of read-ahead, every `read-entry` request that the entry data is required to be read from main storage eventually, will force a read-ahead operation through the method `org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.fillReadAheadCache`. This method will read several entries after the current position and load them into the read-cache, among which the amount of entries is controlled by the `dbStorage_readAheadCacheBatchSize`.
In this mode, once a miss of read-cache occurs, the elapsed time of reading an entry is equivalent to the sum of the time of reading that entry plus reading several entries after the entry, because the process of read-ahead is synchronous.
Synchronous read-ahead is a simple and effective solution in scenarios where read latency is less of a concern. However, we found that when the cluster has a large number of catch-up reads, and the p99 latency cannot be ignored, synchronous read-ahead may introduce a lot of latency glitches. Therefore, we decided to introduce an asynchronous read-ahead mode to reduce the latency for the catch-up reads.
#### Proposed Approach
Instead of modifying the original synchronous read-ahead logic, we introduced an independent asynchronous read-ahead module named `ReadAheadManager`. The user can select a specific read-ahead mode through configuration parameters. The async read-ahead module will provide an interface for reading entry for upper-layer logic.
#### Evaluation Results
Before diving into the details, let's take a look at the performance optimizations brought by asynchronous read-ahead.
> The upgrade time is 2022/3/2 12:00
##### Hit / Miss Count of Read Cache
<img width="1074" alt="image" src="https://user-images.githubusercontent.com/14341827/157023558-a57c520b-8cb8-4aac-9a53-ffb4344a43c3.png">
##### AVG Time of Read-Entry Op
<img width="525" alt="image" src="https://user-images.githubusercontent.com/14341827/157023689-6b3e6b08-3e14-41e3-8602-6af8f8b4dad8.png">
<!-- add a proposal PR link below -->
Proposal PR - #abc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] wuzhanpeng commented on issue #3085: BP-49: Support reading ahead in async mode
Posted by GitBox <gi...@apache.org>.
wuzhanpeng commented on issue #3085:
URL: https://github.com/apache/bookkeeper/issues/3085#issuecomment-1061675830
@merlimat @eolivelli @hangc0276 Hi, could you give some advice?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] StevenLuMT commented on issue #3085: BP-49: Support reading ahead in async mode
Posted by GitBox <gi...@apache.org>.
StevenLuMT commented on issue #3085:
URL: https://github.com/apache/bookkeeper/issues/3085#issuecomment-1084011359
Good job.
I want to know what storage getting this evaluation results, SSD or HDD？
@wuzhanpeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] eolivelli commented on issue #3085: BP-49: Support reading ahead in async mode
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #3085:
URL: https://github.com/apache/bookkeeper/issues/3085#issuecomment-1067692477
next step is to start the discussion on dev@bookkeeper
it looks like a nice work ! well done (I have already taken a quick look to the PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org