You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Avi Flax <av...@aviflax.com> on 2016/01/21 23:02:07 UTC

Forward JMX Metrics to New Relic?

Hello all,

I’m new to Samza and I’m going to be implementing some stream
processing jobs for a client who is also new to Samza. My client uses
New Relic (NR) for metrics and I’d like to get Samza’s metrics
forwarding over to their NR account if possible. I’m just posting here
to get a quick sanity check from someone on what I have in mind. I
don’t really get YARN yet so I could be just completely off in my
thinking.

It’s my understanding that YARN will start a JVM for each
SamzaContainer, which is essentially the application which actually
does most of the processing work. I’m thinking therefore that maybe
each SamzaContainer JVM could load the NR Java Agent[1] via the
`-javaagent` argument, then each job/task (not sure) would publish
metrics via JMX which will then be forwarded to NR as specified in a
local YAML file [2].

Does that make any sense? Does it seems reasonable? Is there a better way?

I’d really appreciate any help with this!

Thanks,
Avi

[1] https://docs.newrelic.com/docs/agents/java-agent/getting-started/new-relic-java

[2] https://docs.newrelic.com/docs/agents/java-agent/custom-instrumentation/custom-jmx-instrumentation-yaml

Re: Forward JMX Metrics to New Relic?

Posted by Avi Flax <av...@aviflax.com>.
> On Jan 21, 2016, at 18:45, Rad Gruchalski <ra...@gruchalski.com> wrote:
> 
> You can make samza publish metrics to a kafka topic using your own serdes -> https://samza.apache.org/learn/documentation/0.9/container/metrics.html  
> From there it should be rather straightforward to consume and put stuff in NR.

Thanks Rad! That makes a ton of sense and seems much simpler than what I was thinking. I'll give it a try and report back!

Thanks!
Avi

Re: Forward JMX Metrics to New Relic?

Posted by Avi Flax <av...@aviflax.com>.
On Fri, Jan 22, 2016 at 1:27 PM Rad Gruchalski <ra...@gruchalski.com> wrote:

I wouldn’t think there’s any reason to use any specific serde other than
> avoiding costly json traversal. Serdes are quite useful for transporting
> typed data.
> You can use them but don’t have to, however, they’re quite helpful.


Got it — that makes sense. Thank you!

Re: Forward JMX Metrics to New Relic?

Posted by Rad Gruchalski <ra...@gruchalski.com>.
Avi,

I wouldn’t think there’s any reason to use any specific serde other than avoiding costly json traversal. Serdes are quite useful for transporting typed data.
You can use them but don’t have to, however, they’re quite helpful.










Kind regards,

Radek Gruchalski

radek@gruchalski.com (mailto:radek@gruchalski.com)
 (mailto:radek@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.



On Friday, 22 January 2016 at 16:09, Avi Flax wrote:

> On Thu, Jan 21, 2016 at 6:45 PM, Rad Gruchalski <radek@gruchalski.com (mailto:radek@gruchalski.com)> wrote:
> > You can make samza publish metrics to a kafka topic using your own serdes…
>  
>  
> Is there any specific reason I should use my own serdes rather than
> just using the default JSON serde that ships with Samza?
>  
> Thanks!
> Avi
>  
>

Re: Forward JMX Metrics to New Relic?

Posted by Avi Flax <av...@aviflax.com>.
On Thu, Jan 21, 2016 at 6:45 PM, Rad Gruchalski <ra...@gruchalski.com> wrote:
> You can make samza publish metrics to a kafka topic using your own serdes…

Is there any specific reason I should use my own serdes rather than
just using the default JSON serde that ships with Samza?

Thanks!
Avi

Re: Forward JMX Metrics to New Relic?

Posted by Rad Gruchalski <ra...@gruchalski.com>.
Avi,  

You can make samza publish metrics to a kafka topic using your own serdes -> https://samza.apache.org/learn/documentation/0.9/container/metrics.html  
From there it should be rather straightforward to consume and put stuff in NR.










Kind regards,

Radek Gruchalski

radek@gruchalski.com (mailto:radek@gruchalski.com)
 (mailto:radek@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.



On Thursday, 21 January 2016 at 23:02, Avi Flax wrote:

> Hello all,
>  
> I’m new to Samza and I’m going to be implementing some stream
> processing jobs for a client who is also new to Samza. My client uses
> New Relic (NR) for metrics and I’d like to get Samza’s metrics
> forwarding over to their NR account if possible. I’m just posting here
> to get a quick sanity check from someone on what I have in mind. I
> don’t really get YARN yet so I could be just completely off in my
> thinking.
>  
> It’s my understanding that YARN will start a JVM for each
> SamzaContainer, which is essentially the application which actually
> does most of the processing work. I’m thinking therefore that maybe
> each SamzaContainer JVM could load the NR Java Agent[1] via the
> `-javaagent` argument, then each job/task (not sure) would publish
> metrics via JMX which will then be forwarded to NR as specified in a
> local YAML file [2].
>  
> Does that make any sense? Does it seems reasonable? Is there a better way?
>  
> I’d really appreciate any help with this!
>  
> Thanks,
> Avi
>  
> [1] https://docs.newrelic.com/docs/agents/java-agent/getting-started/new-relic-java
>  
> [2] https://docs.newrelic.com/docs/agents/java-agent/custom-instrumentation/custom-jmx-instrumentation-yaml