You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/06/17 04:41:16 UTC

[GitHub] [rocketmq-connect] ltamber opened a new issue, #173: Connect Exporter for Prometheus

ltamber opened a new issue, #173:
URL: https://github.com/apache/rocketmq-connect/issues/173

   Currently rocketmq-connect can only get metrics by restful api or cli command, the connect observability can only be used by developers, and the experience is not good. In order to improve the connect observability, the connect metrics should be displayed in Prometheus to improve the connect observability.
   
   -------------
   
   目前RocketMQ Connect只能通过直接调用RESTful接口或者通过cli客户端命令对Connector查看metrics信息，connect可观测性只能是开发同学使用，使用体验不好，为提升connect的可观测性，将connect metrics信息展示在Prometheus，提升connect可观察性


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] odbozhou commented on issue #173: Connect Exporter for Prometheus

Posted by GitBox <gi...@apache.org>.
odbozhou commented on issue #173:
URL: https://github.com/apache/rocketmq-connect/issues/173#issuecomment-1168131445

   Prometheus Sink Connector本身也是一个exporter，只不过是更加通用的exporter。connect把自身的metrics发送到rocketmq的topic当中，Prometheus Sink Connector通过配置把topic里的metrics信息拉去展示在Prometheus当中，其它在组件或者产品例如RocketMQ也可以把自身的metrics发送到topic当中，这样相当于实现一个通用的exporter，可以对接任意metrics信息，而不用再单独实现Prometheus exporter，简化接入Prometheus，只需要把metrics信息发到topic即可。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] sunxiaojian commented on issue #173: Connect Exporter for Prometheus

Posted by GitBox <gi...@apache.org>.
sunxiaojian commented on issue #173:
URL: https://github.com/apache/rocketmq-connect/issues/173#issuecomment-1174959654

   > 
   
   
   
   > > > Prometheus Sink Connector本身也是一个exporter，只不过是更加通用的exporter。connect把自身的metrics发送到rocketmq的topic当中，Prometheus Sink Connector通过配置把topic里的metrics信息拉去展示在Prometheus当中，其它在组件或者产品例如RocketMQ也可以把自身的metrics发送到topic当中，这样相当于实现一个通用的exporter，可以对接任意metrics信息，而不用再单独实现Prometheus exporter，简化接入Prometheus，只需要把metrics信息发到topic即可。
   > > 
   > > 
   > > 当前RocketMQ, RocketMQ Connect 都将metrics写到了日志里面。RocketMQ Connect在Source端也可以实现对SFTP的支持，这样通配置过两个Connector任务，就可以将RocketMQ相关的Metrics信息都接入Prometheus。
   > 
   > 对于您的想法，我有以下疑问
   > 
   > 1. 如果输入的信息从日志中获取，就好像是一个为某种特定日志进行可视化的手段，这样是不是太依赖日志的输出了？对于rocketmq-connect，目前我仅在connect-default.log中发现metrics信息，似乎对于runtime没有专门的metrics日志，那么就会传输许多冗余信息，增加解析时的复杂度。rocketmq确有metrics日志。
   > 2. 日志是追加写的，那么task便存在状态，需要保存每次已经从日志中获取metrics的offset。似乎不是太合适，一旦connector重启，状态就会丢失。我观察了[rocketmq-exporter](https://github.com/apache/rocketmq-exporterhttps://github.com/apache/rocketmq-exporter)的实现，他们似乎是在向rocketmq动态请求相关数据再进行展示。
   > 
   > 我目前有一个不成熟的想法，就是修改runtime，使其在运行过程中将metrics发送至某一topic，然后实现sink connector将数据导出。但这样的通用性却很差。
   
   可以这样做，但是不一定要声明数据写到哪里，我觉得只声明标准的api实现就可以，允许用户自定义exporter,   再connect运行的时候把这些exporter加载进来就可以，这样可以上报到topic ,也可以上报到数据库，可以上报到用户需要的任何地方


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] yhx-coder commented on issue #173: Connect Exporter for Prometheus

Posted by GitBox <gi...@apache.org>.
yhx-coder commented on issue #173:
URL: https://github.com/apache/rocketmq-connect/issues/173#issuecomment-1168717849

   从系统整体的角度，我认为您说的合理。我准备沿着这个方向做一些尝试。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] yhx-coder commented on issue #173: Connect Exporter for Prometheus

Posted by GitBox <gi...@apache.org>.
yhx-coder commented on issue #173:
URL: https://github.com/apache/rocketmq-connect/issues/173#issuecomment-1189966744

   @odbozhou 老师您好，请问sftp source connector 是否设计完毕呢？我想了解其 ConnectRecord 设计，以帮助我进一步完成功能（信息解析与缓存）。目前没有在ISSUE上看到相关进展。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] yhx-coder commented on issue #173: Connect Exporter for Prometheus

Posted by GitBox <gi...@apache.org>.
yhx-coder commented on issue #173:
URL: https://github.com/apache/rocketmq-connect/issues/173#issuecomment-1165541389

   我认为metrics信息对精确性要求不高，无需利用rocketmq来保证消息不丢失。实现一个sink connector 需要引入rocketmq，可能会变得复杂，是不是可以直接实现一个prometheus 的exporter？
   我的水平有限，且刚开始接触rocketmq及connector，给大家一个思路，供大家探讨。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] yhx-coder commented on issue #173: Connect Exporter for Prometheus

Posted by GitBox <gi...@apache.org>.
yhx-coder commented on issue #173:
URL: https://github.com/apache/rocketmq-connect/issues/173#issuecomment-1158504704

   Hi, @[ltamber](https://github.com/ltamber) I am interested in this project.And I`d like to contribute  to the RocketMQ Connect.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] odbozhou commented on issue #173: Connect Exporter for Prometheus

Posted by GitBox <gi...@apache.org>.
odbozhou commented on issue #173:
URL: https://github.com/apache/rocketmq-connect/issues/173#issuecomment-1169457445

   > Prometheus Sink Connector本身也是一个exporter，只不过是更加通用的exporter。connect把自身的metrics发送到rocketmq的topic当中，Prometheus Sink Connector通过配置把topic里的metrics信息拉去展示在Prometheus当中，其它在组件或者产品例如RocketMQ也可以把自身的metrics发送到topic当中，这样相当于实现一个通用的exporter，可以对接任意metrics信息，而不用再单独实现Prometheus exporter，简化接入Prometheus，只需要把metrics信息发到topic即可。
   
   当前RocketMQ, RocketMQ Connect 都将metrics写到了日志里面。RocketMQ Connect在Source端也可以实现对SFTP的支持，这样通配置过两个Connector任务，就可以将RocketMQ相关的Metrics信息都接入Prometheus。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] odbozhou commented on issue #173: Connect Exporter for Prometheus

Posted by GitBox <gi...@apache.org>.
odbozhou commented on issue #173:
URL: https://github.com/apache/rocketmq-connect/issues/173#issuecomment-1173282535

   > 
   1、connect有单独的metrics信息日志，connect_stats.log，可以看下日志的配置是否有问题。
   ![image](https://user-images.githubusercontent.com/14222167/177073007-657a6883-e93a-4e19-9dea-54133195efb9.png)
   从日志获取metrics信息，需要实现sftp source connector，目前还不支持，以后支持。
   2、task是无状态的，可以再任意一个节点上运行。source connector会自动维护日志位点
   3、metrics发送至某一topic可以作为临时方案，以后通过sftp source connector可作为通用方案
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] odbozhou commented on issue #173: Connect Exporter for Prometheus

Posted by GitBox <gi...@apache.org>.
odbozhou commented on issue #173:
URL: https://github.com/apache/rocketmq-connect/issues/173#issuecomment-1165135264

   我有一个想法，是否可以实现一个Connector，Prometheus Sink Connector。完成metrics信息到Prometheus


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq-connect] yhx-coder commented on issue #173: Connect Exporter for Prometheus

Posted by GitBox <gi...@apache.org>.
yhx-coder commented on issue #173:
URL: https://github.com/apache/rocketmq-connect/issues/173#issuecomment-1169970591

   > > Prometheus Sink Connector本身也是一个exporter，只不过是更加通用的exporter。connect把自身的metrics发送到rocketmq的topic当中，Prometheus Sink Connector通过配置把topic里的metrics信息拉去展示在Prometheus当中，其它在组件或者产品例如RocketMQ也可以把自身的metrics发送到topic当中，这样相当于实现一个通用的exporter，可以对接任意metrics信息，而不用再单独实现Prometheus exporter，简化接入Prometheus，只需要把metrics信息发到topic即可。
   > 
   > 当前RocketMQ, RocketMQ Connect 都将metrics写到了日志里面。RocketMQ Connect在Source端也可以实现对SFTP的支持，这样通配置过两个Connector任务，就可以将RocketMQ相关的Metrics信息都接入Prometheus。
   
   对于您的想法，我有以下疑问
   1. 如果输入的信息从日志中获取，就好像是一个为某种特定日志进行可视化的手段，这样是不是太依赖日志的输出了？对于rocketmq-connect，目前我仅在connect-default.log中发现metrics信息，似乎对于runtime没有专门的metrics日志，那么就会传输许多冗余信息，增加解析时的复杂度。rocketmq确有metrics日志。
   2. 日志是追加写的，那么task便存在状态，需要保存每次已经从日志中获取metrics的offset。似乎不是太合适，一旦connector重启，状态就会丢失。我观察了[rocketmq-exporter](https://github.com/apache/rocketmq-exporterhttps://github.com/apache/rocketmq-exporter)的实现，他们似乎是在向rocketmq动态请求相关数据再进行展示。
   
   我目前有一个不成熟的想法，就是修改runtime，使其在运行过程中将metrics发送至某一topic，然后实现sink connector将数据导出。但这样的通用性却很差。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org