[jira] [Created] (NIFI-9273) Metrics: Allow and provice processor custom metrics | refactor metric system

["Theo Diefenthal (Jira)" <ji...@apache.org>]

Theo Diefenthal created NIFI-9273:
-------------------------------------

             Summary: Metrics: Allow and provice processor custom metrics | refactor metric system
                 Key: NIFI-9273
                 URL: https://issues.apache.org/jira/browse/NIFI-9273
             Project: Apache NiFi
          Issue Type: Wish
          Components: Core Framework
            Reporter: Theo Diefenthal


In my current (beginner) understanding of NiFi, each processor reports the very same metrics like processing time, received bytes/files, sent bytes/files. No processor specific metrics exist.

I argue that at least most of the source processors should have their own custom metrics which would drastically improve the user experience when operating NiFi on different agents.

An example: Let's say we have a ConsumeKafka Source connected to a PutKafka Sink mirroring data from a remote kafka to a (nifi-)local kafka cluster. Via NiFi monitoring, we could find backpressure within NiFi, but we wouldn't see if the consumer doesn't catch up reading the messages. We'd thus need additionally tooling to monitor the remote kafka cluster. Same goes e.g. for the GetFile processor where it would be substantial to know the current number of files in the input directory in order to know whether some back pressure builds up. Of course, that doesn't sound like a realistic scenario (Reading from disk would be slower than writing new files) but image reading from a network drive where log files are written local but received via the network.

As a processor developer, I'd also like to include my own custom metrics. From my understanding, I currently can only add deltas to counter types. Gauges thus can't be maintained.

In many applications, thanks to frameworks like Dropwizard or MicroMeter, adding and updating metrics is as easy as writing logging statements. Applications like Spark or Flink don't utilize those libraries but do their best to provide similar features which let you easily define your own metrics anywhere you write user-code and which include metrics from all source components, e.g. when writing to kafka, report all possible kafka-producer-metrics from the kafkaProducer java library. In NiFi, I'm missing those metrics.

In NiFi, a ReportingTask gets some "status" classes about each processor/connection and so on. From that, it builds itself the metrics like Counters and Gauges it wants to report to an external system. In the systems I mentioned earlier, the user can write its own reporter similar to NiFi. The difference is: The reporter doesn't get specific classes like {{{color:#000000}ProcessorStatus}} but instead gets a metric registry with counters, gauges and so on which can then be easily exported to an external system. This way, each processor could add any custom metrics without the need to implement logic for those new metrics in each ReportingTask.
{color}

{color:#000000}Maybe, the Metric handling in NiFi could be inspired by Apache Flink: In Flink, each user code class can receive a "MetricGroup". Those metric groups are kind of pre-configured and properly setup for the task. Let's say in NiFi, we would provide a MetricGroup to some processor. That group would make sure that all metrics added to the group are already labeled with the processor-id and processor-name. And NiFi could as well add all default metrics for this group beforehand, i.e. for each processor something like timeSpent, flowfiles received and so on. {color}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)