[GitHub] [nifi] kevdoran commented on a change in pull request #3519: NIFI-6352: Add ability to report all component metrics to Prometheus

[GitBox <gi...@apache.org>]

kevdoran commented on a change in pull request #3519: NIFI-6352: Add ability to report all component metrics to Prometheus
URL: https://github.com/apache/nifi/pull/3519#discussion_r291724050
 
 

 ##########
 File path: nifi-nar-bundles/nifi-prometheus-bundle/nifi-prometheus-reporting-task/src/main/java/org/apache/nifi/reporting/prometheus/api/PrometheusMetricsUtil.java
 ##########
 @@ -129,6 +147,60 @@
             .labelNames("processor_name", "counter_name", "processor_id", "instance")
             .register(NIFI_REGISTRY);
 
+    // Connection metrics
+    private static final Gauge BACKPRESSURE_BYTES_THRESHOLD = Gauge.build()
+            .name("nifi_backpressure_bytes_threshold")
+            .help("The number of bytes that can be queued before backpressure is applied")
+            .labelNames("instance", "component_type", "component_name", "component_id", "parent_id",
+                    "source_id", "source_name", "destination_id", "destination_name")
+            .register(NIFI_REGISTRY);
+
+    private static final Gauge BACKPRESSURE_OBJECT_THRESHOLD = Gauge.build()
+            .name("nifi_backpressure_object_threshold")
+            .help("The number of flow files that can be queued before backpressure is applied")
+            .labelNames("instance", "component_type", "component_name", "component_id", "parent_id",
+                    "source_id", "source_name", "destination_id", "destination_name")
+            .register(NIFI_REGISTRY);
+
+    private static final Gauge IS_BACKPRESSURE_ENABLED = Gauge.build()
+            .name("nifi_backpressure_enabled")
+            .help("Whether backpressure has been applied for this component. Values are 0 or 1")
+            .labelNames("instance", "component_type", "component_name", "component_id", "parent_id",
+                    "source_id", "source_name", "destination_id", "destination_name")
+            .register(NIFI_REGISTRY);
+
+    // Port metrics
+    private static final Gauge IS_TRANSMITTING = Gauge.build()
+            .name("nifi_transmitting")
+            .help("Whether this component is transmitting data. Values are 0 or 1")
+            .labelNames("instance", "component_type", "component_name", "component_id", "parent_id", "run_status")
+            .register(NIFI_REGISTRY);
+
+    // Remote Process Group (RPG) metrics
+    private static final Gauge ACTIVE_REMOTE_PORT_COUNT = Gauge.build()
+            .name("nifi_active_remote_port_count")
+            .help("The number of active remote ports associated with this component")
+            .labelNames("instance", "component_type", "component_name", "component_id", "parent_id",
+                    "source_id", "source_name", "destination_id", "destination_name")
+            .register(NIFI_REGISTRY);
+
+    private static final Gauge INACTIVE_REMOTE_PORT_COUNT = Gauge.build()
+            .name("nifi_inactive_remote_port_count")
+            .help("The number of inactive remote ports associated with this component")
+            .labelNames("instance", "component_type", "component_name", "component_id", "parent_id",
+                    "source_id", "source_name", "destination_id", "destination_name")
+            .register(NIFI_REGISTRY);
+
+    private static final Gauge AVERAGE_LINEAGE_DURATION = Gauge.build()
 
 Review comment:
   Eventually, it would be good (if possible) to change the components to output two values that can be used to compute  average_lineage_duration rather than (or in addition to) pre-computing it, i.e.:
   
   1. Count of flow files processed (already we are getting this)
   2. Total (aggregate/sum) duration for all flow files processed
   
   With these two values, the consumer of the metrics can, at query time, decide the look back time interval over which to compute the average, and also get the throughput over a time interval. For example (using prometheus query expression syntax):
   
   ```
   # average lineage duration over a 30s look back window
   rate(ff_timer_sum[30s])/rate(ff_timer_count[30s])
   
   # throughput (flow files per second) over a 20s look back window
   rate(ff_timer_count[20s])
   ```
   
   I think this can wait for now as it appears it would require some changes to the underlying Status objects in order to surface the total/sum time.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services