[rocketmq] branch develop updated: [RIP-9] Add english document of storage design(#809)

[di...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dinglei pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/rocketmq.git


The following commit(s) were added to refs/heads/develop by this push:
     new 648cac3  [RIP-9] Add english document of storage design(#809)
648cac3 is described below

commit 648cac3eaddafd54016f685084c7b50aa583c723
Author: vonzhou <vo...@163.com>
AuthorDate: Thu Feb 21 16:48:29 2019 +0800

    [RIP-9] Add english document of storage design(#809)
    
    [RIP-9] Add english document of storage design
---
 docs/en/Design_Store.md                   |  36 ++++++++++++++++++++++++++++++
 docs/en/images/rocketmq_storage_arch.png  | Bin 0 -> 111170 bytes
 docs/en/images/rocketmq_storage_flush.png | Bin 0 -> 22281 bytes
 3 files changed, 36 insertions(+)

diff --git a/docs/en/Design_Store.md b/docs/en/Design_Store.md
new file mode 100644
index 0000000..ac14022
--- /dev/null
+++ b/docs/en/Design_Store.md
@@ -0,0 +1,36 @@
+# Message Storage
+
+
+![](images/rocketmq_storage_arch.png)
+
+Message storage is the most complicated and important part of RocketMQ. This section will describe the three aspects of RocketMQ: message storage architecture, PageCache and memory mapping, and RocketMQ's two different disk flushing methods.
+
+## 1. Message Storage Architecture
+
+
+The message storage architecture diagram consists of 3 files related to message storage: `CommitLog` file, `ConsumeQueue` file, and `IndexFile`.
+
+
+* `CommitLog`：The `CommitLog` file stores message body and metadata sent by producer, and the message content is not fixed length. The default size of one `CommitLog` file is 1G, the length of the file name is 20 digits, the left side is zero padded, and the remaining is the starting offset. For example, `00000000000000000000` represents the first file, the starting offset is 0, and the file size is 1G=1073741824, when the first `CommitLog` file is full, the second `CommitLog` file is `0 [...]
+* `ConsumeQueue`: The `ConsumeQueue` is used to improve the performance of message consumption. Since RocketMQ uses topic-based subscription mode, message consumption is specific to the topic. Traversing the commitlog file to retrieve messages of one topic is very inefficient. The consumer can find the messages to be consumed according to the `ConsumeQueue`. The `ConsumeQueue`(logic consume queue) as an index of the consuming message stores the starting physical offset `offset` in `Commi [...]
+* `IndexFile`: The `IndexFile` provides a way to query messages by key or time interval. The path of the `IndexFile` is `$HOME/store/index/${fileName}`, the file name `fileName` is named after the timestamp when it was created. One IndexFile's size is about 400M, and it can store 2000W indexes. The underlying storage of `IndexFile` is designed to implement the `HashMap` structure in the file system, so RocketMQ's index file is a hash index.
+
+
+From the above architecture of the RocketMQ message storage, we can see RocketMQ uses a hybrid storage structure, that is, all the queues in an instance of the broker share a single log file `CommitLog` to store messages. RocketMQ's hybrid storage structure(messages of multiple topics are stored in one CommitLog) uses a separate storage structure for the data and index parts for Producer and Consumer respectively. The Producer sends the message to the Broker, then the Broker persists the [...]
+
+## 2 PageCache and memory map
+
+PageCache is a cache of files by the operating system to speed up the reading and writing of files. In general, the speed of sequential read and write files is almost the same as the speed of read and write memory. The main reason is that the OS uses a portion of the memory as PageCache to optimize the performance of the read and write operations. For data writing, the OS will first write to the Cache, and then the `pdflush` kernel thread asynchronously flush the data in the Cache to the [...]
+
+In RocketMQ, the logic consumption queue `ConsumeQueue` stores less data and is read sequentially. With the help of prefetch of the page cache mechanism, the read performance of the `ConsumeQueue` file is almost close to the memory read, even in the case of message accumulation, it does not affect performance. But for the log data file `CommitLog`, it will generate many random access reads when reading the message content, which seriously affects the performance. If you choose the approp [...]
+
+
+In addition, RocketMQ mainly reads and writes files through `MappedByteBuffer`. `MappedByteBuffer` uses the `FileChannel` model in NIO to directly map the physical files on the disk to the memory address in user space (`Mmap` method reduces the performance overhead of traditional IO copying disk file data back and forth between the buffer in kernel space and the buffer in user space), it converts the file operation into direct memory address manipulation, which greatly improves the effic [...]
+
+## 3 Message Disk Flush
+
+![](images/rocketmq_storage_flush.png)
+
+
+* synchronous flush: As shown above, the RocketMQ's Broker will return a successful `ACK` response to the Producer after the message is truly persisted to disk. Synchronous flushing is a good guarantee for the reliability of MQ messages, but it will have a big impact on performance. Generally, it is suitable for financial business applications.
+* asynchronous flush: Asynchronous flushing can take full advantage of the PageCache of the OS, as long as the message is written to the PageCache, the successful `ACK` can be returned to the Producer. The message flushing is performed by the background asynchronous thread, which reduces the read and write delay and improves the performance and throughput of the MQ.
\ No newline at end of file
diff --git a/docs/en/images/rocketmq_storage_arch.png b/docs/en/images/rocketmq_storage_arch.png
new file mode 100644
index 0000000..8c719e1
Binary files /dev/null and b/docs/en/images/rocketmq_storage_arch.png differ
diff --git a/docs/en/images/rocketmq_storage_flush.png b/docs/en/images/rocketmq_storage_flush.png
new file mode 100644
index 0000000..1610ae0
Binary files /dev/null and b/docs/en/images/rocketmq_storage_flush.png differ