DataDog reporting task is not working AWS EKS

[sanjeet rath <ra...@gmail.com>]

Hi ,

My 3 node nifi cluster is running on AWS EKS.& for monitoring i have added
Datadog reporting task.
Our structure is data-agent is running as demonSet pod inside the host. and
from nifi pod the datadog reporting task is sending the metrics to the
dd_agent_host over UDP in 8125 port.

i saw the bellow code will work when the dd agent is installed in local
pod/instance only.So i build a custom reporting task with bellow
modification.
Code Reference: *nifi-1.12.1* <https://github.com/coco11563/nifi-1.4.0>/
nifi-nar-bundles
<https://github.com/coco11563/nifi-1.4.0/tree/master/nifi-nar-bundles>/
nifi-datadog-bundle
<https://github.com/coco11563/nifi-1.4.0/tree/master/nifi-nar-bundles/nifi-datadog-bundle>
/nifi-datadog-reporting-task
<https://github.com/coco11563/nifi-1.4.0/tree/master/nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-reporting-task>
/src
<https://github.com/coco11563/nifi-1.4.0/tree/master/nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-reporting-task/src>
/main
<https://github.com/coco11563/nifi-1.4.0/tree/master/nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-reporting-task/src/main>
/java
<https://github.com/coco11563/nifi-1.4.0/tree/master/nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-reporting-task/src/main/java>
/org
<https://github.com/coco11563/nifi-1.4.0/tree/master/nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-reporting-task/src/main/java/org>
/apache
<https://github.com/coco11563/nifi-1.4.0/tree/master/nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-reporting-task/src/main/java/org/apache>
/nifi
<https://github.com/coco11563/nifi-1.4.0/tree/master/nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-reporting-task/src/main/java/org/apache/nifi>
/reporting
<https://github.com/coco11563/nifi-1.4.0/tree/master/nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-reporting-task/src/main/java/org/apache/nifi/reporting>
/datadog
<https://github.com/coco11563/nifi-1.4.0/tree/master/nifi-nar-bundles/nifi-datadog-bundle/nifi-datadog-reporting-task/src/main/java/org/apache/nifi/reporting/datadog>
/*DDMetricRegistryBuilder.java*

private DatadogReporter createDatadogReporter(MetricRegistry
metricRegistry) throws IOException
{
DatadogReporter reporter = DatadogReporter.forRegistry(metricRegistry)
.withHost(InetAddress.getLocalHost().getHostName())  // My new
code .withHost(System.getenv("env variable for dd-agent host"))
.withTransport(transport) .withTags(tags) .build(); return reporter;
}

So the above modified code will get the dd-agent-host from env variable ,
which is host address of the demonset pod where dd-agent is running.
few log i have added where i can see nifi metrics are created with the
configs that i have provided in the reporting task set up(in nifi-app.log
file)
However i am not able to see the nifi metrics in datadog UI.

For testing i did execute the below code from the pod where nifi is running
echo -n "test:custom_metric:60|g|" >/dev/udp/$"env variable for dd-agent
host"/8125
I am able to see the test metrics in datadog UI ,which confirm the
connection is fine.

Could someone please help me to understand any thing i am doing wrong or
missing.

Regards
-- 
Sanjeet Kumar Rath,