[GitHub] [skywalking] junneyang commented on issue #3435: Is it necessary to add MQ (Kafka or RabbitMQ) in skywalking ?

[GitBox <gi...@apache.org>]

junneyang commented on issue #3435: Is it necessary to add MQ (Kafka or RabbitMQ) in skywalking ?
URL: https://github.com/apache/skywalking/issues/3435#issuecomment-606364876
 
 
   > Let's talk about this, since a lot of people used to ask about this before.
   > 
   > There are two parts of questions related to MQ.
   > 
   > # Is MQ a good or right way to community with OAP backend?
   > This question comes out when people think about what happens when the OAP cluster is not powerful enough or offline.
   > But I want to ask the questions before answer this.
   > 
   > 1. Why do you think OAP should be not powerful enough? As it is not, the speed of data analysis wouldn't catch up with producers(agents). Then what is the point of adding new deployment requirement?
   > 2. Maybe you will argue says, the payload is sometimes higher than usual as there is hot business time. But, my question is how much higher?
   > 3. If less than 40%, how many resources will you use for the new MQ cluster? How about moving them to new OAP and ES nodes?
   > 4. If higher than 40%, 70%~2x? Then, I could say, your MQ wastes more resources than it saves. Your MQ would support 2x~3x payload, and with 10%-20% cost in general time. Furthermore, in this case, if the payload/throughput are so high, how long the OAP cluster could catch up. I would say never before it catches up, the next hot time event is coming.
   > 
   > Besides all this analysis, why do you want the traces still 100%, as you are costing so many resources? Better than this, we could consider adding a better dynamic trace sampling mechanism at the backend, when throughputs go over the threshold, active the sampling rate to 100%->10% step by step, which means you could get the OAP and ES 3 times more powerful than usual, just ignore the traces at hot time.
   > 
   > # How about MQ metrics data exporter?
   > I would say, it is already available there. Exporter module with gRPC default mechanism is there. It is easy to provide a new implementor of that module.
   > 
   > Here are my thoughts, hope it helps.
   
   一句话，不支持MQ，意味着整个平台是封闭的！

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services