[GitHub] [inlong] healchow commented on pull request #6227: [INLONG-6226][Manager] Add a consumer group of TubeMQ for each sink

[GitBox <gi...@apache.org>]

healchow commented on PR #6227:
URL: https://github.com/apache/inlong/pull/6227#issuecomment-1286826235

   Currently, for TubeMQ, we only create a consumer group at the InlongGroup level, consume a piece of data and write it to various Sinks (such as Hive and ES) under different Streams.
   
   Since the topic of TubeMQ is at the level of InlongGroup, Stream is just a message of different types in the topic (distinguished by the streamId in the message header). Creating only one consumer group can reduce the pressure on the service side of TubeMQ to save a large amount of consumer group information, and also avoid repeated consumption of a large amount of data.
   
   
   当前，对于TubeMQ，我们只在 InlongGroup 层面创建一个消费组，消费1份数据并写到不同的Stream下的各类 Sink （如Hive、ES）中。
   
   由于 TubeMQ 的 Topic 就在 InlongGroup 层面，Stream 只是Topic中的不同类型的消息（由消息头中的streamId来区分），只创建1个消费组就能减少TubeMQ的服务端保存大量消费组信息的压力，也能避免重复消费大量的数据。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org