You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@rocketmq.apache.org by Gosling Von <fe...@gmail.com> on 2018/11/06 09:41:32 UTC
[DISCUSS] [RIP] Message Track Trace
+1
Very cool, It seems to the fifth RIP from the community.
Best Regards,
Von Gosling
> 在 2018年11月6日，下午1:52，huzongtang@cmss.chinamobile.com 写道：
>
> FYI
>
> 发件人： huzongtang@cmss.chinamobile.com <ma...@cmss.chinamobile.com>
> 发送时间： 2018-11-06 13:38
> 收件人： users-subscribe <ma...@rocketmq.apache.org>; commits <ma...@rocketmq.apache.org>; dev-help <ma...@rocketmq.apache.org>
> 主题： About the RIP for new Feature：Message Track Trace
> RIP-1: Message Track Trace
>
> 1、Status
> Current State: Proposed
> Authors: HuzongTang
> Shepherds: Zhendong
> Mailing List Discussion: <apache mailing list archive>
> Pull Request: #PR_NUMBER
> Released: <released_version>
> 2、Background & Motivation
> What do we need to do
> Message track trace refers to the consumption processing of a message which is sent from the producer instance has arrived the broker,and then has been consumed by the consumer instance.In the whole process, the information of time, location and other related services(including Client and Broker) is aggregated into the complete link information of message communication.
> In MQ system, the complete link of a message contains three roles: producer, broker server and consumer.In the process of message communicaton, each role adds relevant information to the track trace link.By aggregating these information,it can provide some powerful support to user.
> About the Apache RocketMQ project,I will add the feature of message track trace which can help users query every complete link data of a message exactly.This funcation is important to RocketMQ,especially in some financial application fields.
>
> 3、Goals
> What problem is this proposal designed to solve?
> Reliability and Availability is the most two important characters for each MQ system.Although RocketMQ does very well in the two fields,it need some other methods to make sure the complete link of a message is no problem.By some ways,we should view the complete link of a messag and find the root cause of failure in process messaging quickly.
> So,if RocketMQ support the feature of Message Track Trace,we can find and analyse the root cause of failure in process messaging easily.And we can query many parameter values,such as sending cost time,consumption cost time,store time in broker and so on.The architecture of Message Track Trace will be introduced in below chapter.
>
> 4、Non-Goals
> What problem is this proposal NOT designed to solve?
> Every users who build application by using RocketMQ may lack of a efficient way to find and analyse the root cause of failure in process messaging easily and quickly.The users may spend much more time and enery on operation of RocketMQ in production enviroment.
> Are there any limits of this proposal?
> In order to reduce the impact of message entity content storage in RocketMQ，we redefine a special broker which stores every message track trace.So,if user don't want to use this funcation,he can close it in client by setting a flag.
>
> 5、Changes
> We need add some codes in client and broker component which include collecting producer and consumer instances' infomation and redefine a special broker node.Read below sections to get more details about the Message Track Trace for RocketMQ.
>
> 6、Architecture
> We plan to design the function of message track trace including its store part and client instance collecting.The total architecture has two part below:
> （1）Client instance collects the infomation of message track trace
>
> The design of architecture above as follows:
> （a）."producer and consumer multi-thread module, and Blocking Queue"：here, in client,as a producer model,we can collect the infomation(such as,sending and consuming message cost time,broker store time,broker stored ip address and so on) and put this information into the Blocking Queue.And as s consumer model,use a thread called "MQ-AsyncArrayDispatcher" to take the message track trace infomation from Blocking Queue.Then this asynchronous thread( called "MQ-AsyncArrayDispatcher") pack the message track trace element as AsyncAppenderRequest task and submit to the thead pool.
> Last,the main execution process of AsyncAppenderRequest task is sending the message track trace infomation which is collected above from client side to a special broker node which is redefined in below chapter.
> （b）.define a new pair hook which is implementation of the“SendMessageHook/ConsumeMessageHook”,from this,we can collect message track trace data before and after publishing and subscribing messages.
>
> （2）Redefine a special broker node to store message track trace data
>
> As shown in the above picture,we can define a specail broker service node which can store the message track trace data in a common RocketMQ cluster.Here,we can add a flag(such as autoTraceBrokerEnable) in broker.properties file and use this variable to define Whether this broker is a specail node for storing message track trace data.
> （a）.autoTraceBrokerEnable is false.This indicates this broker is an ordinary node,then "Trace_Topic" will not be created in this node.And it will still follow the original processing flow without any change.
> （b）.autoTraceBrokerEnable is true.This indicates broker is an special node,which stores the message track trace data specailly.And The "Trace_Topic" is automatically created during the Broker's startup phase,this node automatically registers its owned set of topics in memory to nameserver(including Trace_Topic).Thus,in a RocketMQ cluster,there is only a specail broker node which stores message track trace datas.And clients(including publishing and subscribing messages) knows which broker node to send message trace data after they try fetch topic routing info from nameserver.
>
> （3）How to solve the query message track trace data by original topic For example,Topic which saves message track trace data is called "RMQ_SYS_TRACE_DATA_XXX" is instead of Topic of original data.But，message query(By messageId + topic,topic+ key or topic) which still uses original data on RocketMQ console side can't query the expected result for us.Here, we can fill in the keyset field of the message track trace data by using msgId (not offset MsgId) or Key of the original message content when send the message track trace data collected by the client, so the IndexFile index of the Broker end can be built according to the msgId or Key of the original message.
> And at the broker side,we can invokes the queryMessage() method of the storage part through the QueryMessageProcessor Business processor.
>
> （4）Finishing querying messages track trace datas in console
> Fisrtly,the console project is based on spring boot technology,so we can finish querying messages track track datas in this project.The basic design is using thread pool to send a RPC request to the redefined broker node to get the message track trace data.
>
> 7、Interface Design/Change
> Here I just list part interface desgin and change in Client Side.And the chang parts of Broker and Console side leave out.
> （1）The data transfering asynchronously interface is as followers:
> public interface AsyncDispatcher {
>
> void start() throws MQClientException;
>
> boolean append(Object ctx);
>
> void flush() throws IOException;
>
> void shutdown();
> }
>
> （2）Defining the two Hooks which is implemented of ConsumeMessageHook and SendMessageHook are as followers:
> a.ClientSendMessageTraceHookImpl
> b.ClientConsumeMessageTraceHookImpl
>
> （3）We define and write some codes for common data model which are as followers:
> a.TraceBean
> b.TraceConstants
> c.TraceContext
> d.TraceDataEncoder
> e.TraceDispatcherType
> f.TraceTransferBean
> g.TuxeTraceType
>
> 8.Compatibility, Deprecation, and Migration Plan
> （1）Are backward and forward compatibility taken into consideration?
> No issues;
> （2）Are there deprecated APIs?
> No issues;
> （3）How do we do migration?
> No issues;
>
> 9.Implementation Outline
> We will implement the proposed changes by 1 phase. All of implementation will be finished in 1 phase.
>
> huzongtang@cmss.chinamobile.com <ma...@cmss.chinamobile.com>
Re: [DISCUSS] [RIP] Message Track Trace
Posted by dongeforever <do...@apache.org>.
+1.
Nice RIP, I would like to offer a help.
Gosling Von <fe...@gmail.com> 于2018年11月6日周二 下午5:41写道：
> +1
>
> Very cool, It seems to the fifth RIP from the community.
>
> Best Regards,
> Von Gosling
>
> 在 2018年11月6日，下午1:52，huzongtang@cmss.chinamobile.com 写道：
>
> FYI
>
>
> *发件人：* huzongtang@cmss.chinamobile.com
> *发送时间：* 2018-11-06 13:38
> *收件人：* users-subscribe <us...@rocketmq.apache.org>; commits
> <co...@rocketmq.apache.org>; dev-help <de...@rocketmq.apache.org>
> *主题：* About the RIP for new Feature：Message Track Trace
> RIP-1: Message Track Trace
>
> *1、Status*
>
> - * Current State: Proposed*
> - * Authors: HuzongTang*
> - * Shepherds: Zhendong*
> - Mailing List Discussion: <apache mailing list archive>
> - Pull Request: #PR_NUMBER
> - Released: <released_version>
>
> *2、Background & Motivation*
> What do we need to do
> Message track trace refers to the consumption processing of a
> message which is sent from the producer instance has arrived the broker,and
> then has been consumed by the consumer instance.In the whole process, the
> information of time, location and other related services(including Client
> and Broker) is aggregated into the complete link information of message
> communication.
> In MQ system, the complete link of a message contains three roles:
> producer, broker server and consumer.In the process of message
> communicaton, each role adds relevant information to the track trace
> link.By aggregating these information,it can provide some powerful support
> to user.
> About the Apache RocketMQ project,I will add the feature of message
> track trace which can help users query every complete link data of a
> message exactly.This funcation is important to RocketMQ,especially in
> some financial application fields.
>
> *3、Goals*
>
> - What problem is this proposal designed to solve?
>
> Reliability and Availability is the most two important characters for
> each MQ system.Although RocketMQ does very well in the two fields,it need
> some other methods to make sure the complete link of a message is no
> problem.By some ways,we should view the complete link of a messag and
> find the root cause of failure in process messaging quickly.
> So,if RocketMQ support the feature of Message Track Trace,we can find
> and analyse the root cause of failure in process messaging easily.And we
> can query many parameter values,such as sending cost time,consumption
> cost time,store time in broker and so on.The architecture of Message Track
> Trace will be introduced in below chapter.
>
> *4、Non-Goals*
>
> - What problem is this proposal NOT designed to solve?
>
> Every users who build application by using RocketMQ may lack of
> a efficient way to find and analyse the root cause of failure in process
> messaging easily and quickly.The users may spend much more time and enery
> on operation of RocketMQ in production enviroment.
>
> - Are there any limits of this proposal?
>
> In order to reduce the impact of message entity content storage
> in RocketMQ，we redefine a special broker which stores every message track
> trace.So,if user don't want to use this funcation,he can close it in client
> by setting a flag.
>
> *5、Changes*
> We need add some codes in client and broker component which include
> collecting producer and consumer instances' infomation and redefine a
> special broker node.Read below sections to get more details about the
> Message Track Trace for RocketMQ.
>
> *6、Architecture*
> We plan to design the function of message track trace including its
> store part and client instance collecting.The total architecture has two
> part below:
> * （1）Client instance collects the infomation of message track trace*
>
> The design of architecture above as follows:
> *（a）*."producer and consumer multi-thread module, and Blocking
> Queue"：here, in client,as a producer model,we can collect the infomation(such
> as,sending and consuming message cost time,broker store time,broker stored
> ip address and so on) and put this information into the Blocking Queue.And
> as s consumer model,use a thread called "MQ-AsyncArrayDispatcher" to take
> the message track trace infomation from Blocking Queue.Then this asynchronous
> thread( called "MQ-AsyncArrayDispatcher") pack the message track trace
> element as AsyncAppenderRequest task and submit to the thead pool.
> Last,the main execution process of AsyncAppenderRequest task is
> sending the message track trace infomation which is collected above from
> client side to a special broker node which is redefined in below chapter.
> *（b）*.define a new pair hook which is implementation of the“SendMessageHook/ConsumeMessageHook”,from
> this,we can collect message track trace data before and after publishing
> and subscribing messages.
>
> * （2）Redefine a special broker node to store message track trace data*
>
> As shown in the above picture,we can define a specail broker service
> node which can store the message track trace data in a common RocketMQ
> cluster.Here,we can add a flag(such as autoTraceBrokerEnable) in
> broker.properties file and use this variable to define Whether this
> broker is a specail node for storing message track trace data.
> *（a）.*autoTraceBrokerEnable is false.This indicates this broker is
> an ordinary node,then "Trace_Topic" will not be created in this node.And
> it will still follow the original processing flow without any change.
> *（b）.*autoTraceBrokerEnable is true.This indicates broker is an
> special node,which stores the message track trace data specailly.And The "Trace_Topic" is automatically created during the Broker's startup phase,this
> node automatically registers its owned set of topics in memory to
> nameserver(including Trace_Topic).Thus,in a RocketMQ cluster,there is
> only a specail broker node which stores message track trace datas.And
> clients(including publishing and subscribing messages) knows which broker
> node to send message trace data after they try fetch topic routing info
> from nameserver.
>
> *（3）How to solve the query message track trace data by original topic*
> For example,Topic which saves message track trace data is called "RMQ_SYS_TRACE_DATA_XXX" is instead of Topic of original data.But，message
> query(By messageId + topic,topic+ key or topic)
> which still uses original data on RocketMQ console side can't query the
> expected result for us.
> Here, we can fill in the keyset field of the message track trace data by using msgId (not offset MsgId) or Key of the original message content when send the message track trace data collected by the client, so the IndexFile index of the Broker end can be built according to the msgId or Key of the original message.
> And at the broker side,we can
> invokes the queryMessage() method of the storage part through the QueryMessageProcessor
> Business processor.
>
> * （4）Finishing querying messages track trace datas in console*
> Fisrtly,the console project is based on spring boot technology,so we
> can finish querying messages track track datas in this project.The basic
> design is using thread pool to send a RPC request to the redefined broker
> node to get the message track trace data.
>
> *7、Interface Design/Change*
> Here I just list part interface desgin and change in Client Side.And
> the chang parts of Broker and Console side leave out.
> * （1）The data transfering asynchronously interface is as followers: *
> public interface AsyncDispatcher {
>
> void start() throws MQClientException;
>
> boolean append(Object ctx);
>
> void flush() throws IOException;
>
> void shutdown();
> }
>
> （2）Defining the two Hooks which is implemented of ConsumeMessageHook
> and SendMessageHook are as followers:
> a.ClientSendMessageTraceHookImpl
> b.ClientConsumeMessageTraceHookImpl
>
> * （3）We define and write some codes for common data model which are as
> followers:*
> a.TraceBean
> b.TraceConstants
> c.TraceContext
> d.TraceDataEncoder
> e.TraceDispatcherType
> f.TraceTransferBean
> g.TuxeTraceType
>
> *8.Compatibility, Deprecation, and Migration Plan*
> （1）Are backward and forward compatibility taken into consideration?
> No issues;
> （2）Are there deprecated APIs?
> No issues;
> （3）How do we do migration?
> No issues;
>
>
> *9.Implementation Outline* We will implement the proposed changes by
> 1 phase. All of implementation will be finished in 1 phase.
>
> ------------------------------
> huzongtang@cmss.chinamobile.com
>
>
>