You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by GitBox <gi...@apache.org> on 2022/04/09 09:54:01 UTC

[GitHub] [flume] tmgstevens opened a new pull request, #358: FLUME-3418 Add support for making metrics available to Prometheus

tmgstevens opened a new pull request, #358:
URL: https://github.com/apache/flume/pull/358

   Flume has a comprehensive metrics system build on MBeans which has exporters to HTTP/JSON and Ganglia.
   
   Adding support for Prometheus, which is the de facto standard in the Kubernetes will open Flume up to a much wider range of deployment scenarios and similarly decouple Flume from the monitoring systems of corporates who now no longer ship Flume.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@flume.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flume] rgoers commented on pull request #358: FLUME-3418 Add support for making metrics available to Prometheus

Posted by GitBox <gi...@apache.org>.
rgoers commented on PR #358:
URL: https://github.com/apache/flume/pull/358#issuecomment-1097622563

   Yes, I meant that Flume will now have required dependencies on 
   
   ```
    <dependency>
         <groupId>io.prometheus</groupId>
         <artifactId>simpleclient</artifactId>
       </dependency>
   
       <dependency>
         <groupId>io.prometheus</groupId>
         <artifactId>simpleclient_servlet</artifactId>
       </dependency>
   ```
   even if the property isn't enabled.  This is one reason I would prefer something like a flume-prometheus module so you only get those dependencies if you include flume-prometheus. But, as you say, we are packaging everything as a zip so putting it in a separate jar probably wouldn't buy anything right now. But hopefully it will in 2.0. Putting it in a separate module now might make that work easier.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@flume.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flume] tmgstevens commented on pull request #358: FLUME-3418 Add support for making metrics available to Prometheus

Posted by GitBox <gi...@apache.org>.
tmgstevens commented on PR #358:
URL: https://github.com/apache/flume/pull/358#issuecomment-1100512148

   I've looked into refactoring the instrumentation package into modules. The challenge comes with https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/MonitoringType.java#L28 which would need to change to be a string rather than a class, and therefore we lose some strong typing.
   
   I can change it, but I'm honestly not sold on the cost / benefit. @rgoers let me know which way you want me to go.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@flume.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flume] rgoers commented on pull request #358: FLUME-3418 Add support for making metrics available to Prometheus

Posted by GitBox <gi...@apache.org>.
rgoers commented on PR #358:
URL: https://github.com/apache/flume/pull/358#issuecomment-1096454270

   So now Prometheus becomes a required dependency. Flume has always supported retrieving stats via HTTP. I've been doing that with Splunk for years. Would it be possible to do this in a flume-prometheus module so it can be optional?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@flume.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flume] tmgstevens commented on pull request #358: FLUME-3418 Add support for making metrics available to Prometheus

Posted by GitBox <gi...@apache.org>.
tmgstevens commented on PR #358:
URL: https://github.com/apache/flume/pull/358#issuecomment-1118642356

   @rgoers are you happy to merge this in please?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@flume.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flume] tmgstevens merged pull request #358: FLUME-3418 Add support for making metrics available to Prometheus

Posted by GitBox <gi...@apache.org>.
tmgstevens merged PR #358:
URL: https://github.com/apache/flume/pull/358


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@flume.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flume] tmgstevens commented on pull request #358: FLUME-3418 Add support for making metrics available to Prometheus

Posted by GitBox <gi...@apache.org>.
tmgstevens commented on PR #358:
URL: https://github.com/apache/flume/pull/358#issuecomment-1097466601

   It's only turned on if you enable it via -Dflume.monitoring.type=prometheus. We don't really have a modular packaging system at the moment, which is a shame, but it does at least mean there's on JAR that does everything.
   The delta for this is there would be no impact for anybody using Ganglia or HTTP, but this adds in Prometheus/OpenMetrics format as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@flume.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flume] rgoers commented on pull request #358: FLUME-3418 Add support for making metrics available to Prometheus

Posted by GitBox <gi...@apache.org>.
rgoers commented on PR #358:
URL: https://github.com/apache/flume/pull/358#issuecomment-1100594422

   Yes, I see the problem. No, you should not change it now. But this is exactly what I would like to change for 2.0. Flume does this is many places and it is too brittle. I would like to adopt a plugin system similar to what Log4j 2 does that allows new components like these to be added dynamically without having to update an enum file. i.e. - PrometheusHttpMetricsServer would be annotated to indicate it is a Plugin of type Metrics server with a name of PROMETHEUS (or the mixed case equivalent) so the loadMonitoring method in Application would be able to find it and start it without requiring an enum. This same stuff goes on for Sources, Sinks, Channels, etc. which makes user provided components second class citizens since they have to be configured via the fully qualified class name.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@flume.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org