You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by aa...@apache.org on 2022/06/29 03:24:29 UTC
[rocketmq-clients] branch master updated: Docs: add design document(WiP) (#18)
This is an automated email from the ASF dual-hosted git repository.
aaronai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/rocketmq-clients.git
The following commit(s) were added to refs/heads/master by this push:
new c7e9250 Docs: add design document(WiP) (#18)
c7e9250 is described below
commit c7e92509e72da5d598f600f875040b6a41a4d4ab
Author: Aaron Ai <ya...@gmail.com>
AuthorDate: Wed Jun 29 11:24:25 2022 +0800
Docs: add design document(WiP) (#18)
---
docs/artwork/client_periodic_task.png | Bin 0 -> 306462 bytes
docs/artwork/client_startup_process.png | Bin 0 -> 471540 bytes
.../fifo_message_group_message_queue_mapping.png | Bin 0 -> 315857 bytes
.../message_consumption_in_push_consumer_FIFO.png | Bin 0 -> 608112 bytes
...ssage_consumption_in_push_consumer_non_FIFO.png | Bin 0 -> 388151 bytes
docs/artwork/message_publishing_in_producer.png | Bin 0 -> 694016 bytes
...ge_receiving_and_process_in_simple_consumer.png | Bin 0 -> 253992 bytes
.../artwork/message_receiving_in_push_consumer.png | Bin 0 -> 402721 bytes
docs/artwork/message_types_and_topic_types.png | Bin 0 -> 598707 bytes
docs/artwork/unified_apis_and_spec.png | Bin 0 -> 119393 bytes
docs/design.md | 76 ++++++++++++++++++
docs/message_id.md | 4 +-
docs/workflow.md | 87 +++++++++++++++++++++
13 files changed, 166 insertions(+), 1 deletion(-)
diff --git a/docs/artwork/client_periodic_task.png b/docs/artwork/client_periodic_task.png
new file mode 100644
index 0000000..2b3a23e
Binary files /dev/null and b/docs/artwork/client_periodic_task.png differ
diff --git a/docs/artwork/client_startup_process.png b/docs/artwork/client_startup_process.png
new file mode 100644
index 0000000..dc6bc9e
Binary files /dev/null and b/docs/artwork/client_startup_process.png differ
diff --git a/docs/artwork/fifo_message_group_message_queue_mapping.png b/docs/artwork/fifo_message_group_message_queue_mapping.png
new file mode 100644
index 0000000..a8fee8f
Binary files /dev/null and b/docs/artwork/fifo_message_group_message_queue_mapping.png differ
diff --git a/docs/artwork/message_consumption_in_push_consumer_FIFO.png b/docs/artwork/message_consumption_in_push_consumer_FIFO.png
new file mode 100644
index 0000000..1efc6a1
Binary files /dev/null and b/docs/artwork/message_consumption_in_push_consumer_FIFO.png differ
diff --git a/docs/artwork/message_consumption_in_push_consumer_non_FIFO.png b/docs/artwork/message_consumption_in_push_consumer_non_FIFO.png
new file mode 100644
index 0000000..8aa1e6f
Binary files /dev/null and b/docs/artwork/message_consumption_in_push_consumer_non_FIFO.png differ
diff --git a/docs/artwork/message_publishing_in_producer.png b/docs/artwork/message_publishing_in_producer.png
new file mode 100644
index 0000000..02ba697
Binary files /dev/null and b/docs/artwork/message_publishing_in_producer.png differ
diff --git a/docs/artwork/message_receiving_and_process_in_simple_consumer.png b/docs/artwork/message_receiving_and_process_in_simple_consumer.png
new file mode 100644
index 0000000..46ee3c3
Binary files /dev/null and b/docs/artwork/message_receiving_and_process_in_simple_consumer.png differ
diff --git a/docs/artwork/message_receiving_in_push_consumer.png b/docs/artwork/message_receiving_in_push_consumer.png
new file mode 100644
index 0000000..e237b63
Binary files /dev/null and b/docs/artwork/message_receiving_in_push_consumer.png differ
diff --git a/docs/artwork/message_types_and_topic_types.png b/docs/artwork/message_types_and_topic_types.png
new file mode 100644
index 0000000..e9fbfed
Binary files /dev/null and b/docs/artwork/message_types_and_topic_types.png differ
diff --git a/docs/artwork/unified_apis_and_spec.png b/docs/artwork/unified_apis_and_spec.png
new file mode 100644
index 0000000..9b471fc
Binary files /dev/null and b/docs/artwork/unified_apis_and_spec.png differ
diff --git a/docs/design.md b/docs/design.md
new file mode 100644
index 0000000..b77ba45
--- /dev/null
+++ b/docs/design.md
@@ -0,0 +1,76 @@
+# Design
+
+This project aims to build lightweight, cloud-native clients, and establish the unified messaging model/APIs design acorss languages.
+
+## Messaging Model
+
+We assume that you are familiar with the basic concepts and terminologies about Apache RocketMQ. Therefore, we will not go into details about that.
+
+### Message/Topic Types
+
+In order to adapt different scenarios, we defined different types for messages to publish.
+
+* **FIFO**(First In, First Out): follows standard [FIFO](https://en.wikipedia.org/wiki/FIFO) semantics, this type of message must set the `message group`. This type of message need to be used with FIFO consumer group, which gurantees that the message must be received first if the it is published first with the same message group.
+* **DELAY**: the message about to be sent is not immediately visible to the consumer, this type of message must set the `delivery timestamp`, which decides the visible time to consumers;
+* **TRANSACTIONAL**: wrap the publishing of one or more messages into a transaction, provide commit/rollback method to decide the visibility of messages;
+* **NORMAL**: default type, which means the message is normal type if it does not belong to the other types above.
+
+The different types are mutually exclusive, when means that the message to publish could not be both FIFO type and DELAY type at the same time. Actually the type of the message is bound to the topic type in, and the type of the topic determines the type of the message. For example, FIFO topic do not allow the publishing of other types of messages.
+
+<div align="center">
+<img src="./artwork/message_types_and_topic_types.png" width="60%">
+</div>
+
+### Consumer Group
+
+As we all know, consumer group is the basic unit of load balancing for consumers. Messages are evenly distributed to all consumers in the same consumer group. Now, we assigned more attributes to consumer group rather than the consumer itself.
+
+* Consume timeout: maximum time window bewteen receiving messages and returning an response of acknowledgement. Message will be visible again if time runs out.
+* FIFO consumption switch: enable/disable FIFO consumption for FIFO messages.
+* Message consumption retry policy: decide the max delivery times and backoff algorithm for the message which is failed to be consumed in push consumer.
+
+Especially, this table shows the sequence of message receiption with the combination of FIFO/non-FIFO topics and FIFO/non-FIFO consumer groups.
+
+<div align="center">
+
+| | FIFO Consumer Group | non-FIFO Consumer Group |
+| :------------: | :-----------------: | :---------------------: |
+| FIFO Topic | FIFO | non-FIFO |
+| non-FIFO Topic | non-FIFO | non-FIFO |
+
+</div>
+
+### Client Categories
+
+There are four types of client.
+
+* Producer: the client to publish message.
+* Push consumer: full-managed consumer. Message listener are provided for consumption.
+* Simple consumer: the simplest consumer, providing methods to receive messages and change invisible time for messages.
+* Pull consumer: for streaming, it allows to get route info manually and select message queues to bind for pulling message.
+
+### Message Identifier
+
+Message identifer provides identity information for each message stored in broker, which means producer can not get the message identifier until it is sent out successfully.
+
+>**Note**: Internal retries during message publishing may cause message duplication, the duplicate messages here have the same message indentifier.
+
+The message identifer layout is redesigned, see more details [here](./message_id.md).
+
+## New and Unified APIs
+
+<div align="center">
+<img src="./artwork/unified_apis_and_spec.png" width="40%">
+</div>
+
+### Original
+
+[RIP-37: New and Unified APIs](https://shimo.im/docs/m5kv92OeRRU8olqX): RocketMQ proposal of new and unified APIs across different languages.
+
+### Background & Rationale
+
+In fact, current rocketmq APIs exist some problems:
+
+* Lack of well-designed messaging models, which is the prerequisite for unified APIs specification. What’s more, the model strongly affects the definition about behavior. Ambiguous definition makes it harder to predict the behavior of the client.
+* Implementation-oriented. It is not friendly to developers. Many features are based on implementation rather than interface. In addition, users are exposed to redundant and complex interfaces with too many details, increasing probability of making mistakes.
+* Without an unified APIs lifecycle management system, deprecated APIs are reserved for a long time, making APIs difficult to be upgraded.
diff --git a/docs/message_id.md b/docs/message_id.md
index aa58cb7..89f02e8 100644
--- a/docs/message_id.md
+++ b/docs/message_id.md
@@ -19,7 +19,9 @@ As for the second type, there are some major flaws in the algorithm design. Firs
## Detailed Design
-
+<div align="center">
+<img src="./artwork/message_id.png" width="70%">
+</div>
This illustration shows an example of message identifier. The version number of the newly designed version is 0x01, and number of all previous versions is 0x00. The message identifier's length is 17 bytes if its version is higher than 0x00, then we convert it to uppercase hexadecimal to get the message identifier in string format.
diff --git a/docs/workflow.md b/docs/workflow.md
new file mode 100644
index 0000000..3fe0eb9
--- /dev/null
+++ b/docs/workflow.md
@@ -0,0 +1,87 @@
+# Workflow
+
+This document elaborates the unified workflow of client. The specific implementations maybe differ from language to language, but they follow the same workflow.
+
+## Startup
+
+Different from previous clients, the new version adds some preparations during the startup. One benefit of this change is to catch more obvious errors or exceptions earlier. These perparations include:
+
+1. Try to fetch route data of topics.
+2. Try to get settings from the server, which could do hot-update about these settings. we call this process **server-client telemetry**.
+
+Failure of any preparation will result in the failure of client startup.
+
+<div align="center">
+<img src="./artwork/client_startup_process.png" width="80%">
+</div>
+
+In details, the **server-client telemetry** provides a channel to upload the local settings and to overwrite the client settings.
+
+## Periodic Task
+
+The client performs same tasks periodically.
+
+* Update topic route data and cache it. The subsequent request could get route from cache directly.
+* Send heartbeart to keep alive.
+* Send **server-client telemetry** request. Client settings may be overwritten by telemetry response.
+
+<div align="center">
+<img src="./artwork/client_periodic_task.png" width="60%">
+</div>
+
+## Message Flow in Producer
+
+The workflow to publish a single message of NORMAL type. The message publishing of other types and publishing of batch messages is similar to it. Some special cases will be explained later.
+
+<div align="center">
+<img src="./artwork/message_publishing_in_producer.png" width="70%">
+</div>
+
+The publishing process is as follows:
+
+1. Check if topic route is cached before or not.
+2. If topic route is not cached, then try to fetch it from server, otherwise go to step 4.
+3. Return failure and end the current process if topic route is failed to fetch, otherwise cache the topic route and go to the next step.
+4. Select writable candicate message queues from topic route to publish meessage.
+5. Return failure and end the current process if the type of message queue is not matched with message type.
+6. Attempt to publish message.
+7. Return success and end the current process if message is published successfully.
+8. Catch the error information of failed message publishing.
+9. Return failure and end the current process if the attempt times is run out, otherwirse deecide to retry or not according to the error type.
+10. Return failure and end the current process if there is no need to retry, otherwise go to the next step.
+11. Isolate the current endpoint for publishing.
+12. Rotate to next message queue to publish message, and go to step 6.
+
+> **Note**: The failure of message publishing will isolate the endpoint, this makes the endpoint will not be selected for load balancing as much as possible. The periodic heartbeat to the isolate endpoint will check health status about it and remove it from the isolate endpoints if no more exception is deteched.
+
+For TRANSACTIONAL messages, the publishing will not be retried if failure is encountered. The ordering of FIFO message is based on the assumption that messages which have the same `message group` will be put into the same message queue, thus the message queue to publish is not disturbed by the isolated enpoint([SipHash](https://en.wikipedia.org/wiki/SipHash) algorithm is used to calculate the message queue index for FIFO message publishing).
+
+<div align="center">
+<img src="./artwork/fifo_message_group_message_queue_mapping.png" width="50%">
+</div>
+
+## Message Flow in Push Consumer
+
+### Message Receiving in Push Consumer
+
+<div align="center">
+<img src="./artwork/message_receiving_in_push_consumer.png" width="70%">
+</div>
+
+### Message Consumption in Push Consumer(Non-FIFO)
+
+<div align="center">
+<img src="./artwork/message_consumption_in_push_consumer_non_FIFO.png" width="50%">
+</div>
+
+### Message Consumption in Push Consumer(FIFO)
+
+<div align="center">
+<img src="./artwork/message_consumption_in_push_consumer_FIFO.png" width="60%">
+</div>
+
+## Message Flow in Simple Consumer
+
+<div align="center">
+<img src="./artwork/message_receiving_and_process_in_simple_consumer.png" width="50%">
+</div>
\ No newline at end of file