You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Vaibhav Garg <va...@gmail.com> on 2020/03/01 06:23:35 UTC
Re: Samza - New Relic integration
I would be really grateful if someone could help me with this.
Thanks,
Vaibhav Garg
+91-9505020924
vaibhavgarg90@gmail.com
LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
On Sat, Feb 29, 2020 at 9:02 PM Vaibhav Garg <va...@gmail.com>
wrote:
> Please do let me know if some other clarification is needed.
>
> Thanks,
> Vaibhav Garg
> +91-9505020924
> vaibhavgarg90@gmail.com
> LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>
>
> On Sat, Feb 29, 2020 at 10:31 AM Vaibhav Garg <va...@gmail.com>
> wrote:
>
>> OK. Here is what I have inside my configuration in order to start
>> NewRelic at the time of the container start:
>>
>> task.opts=-Xmx2048m -XX:+UseConcMarkSweepGC
>> *-javaagent:/usr/home/vaibhav/newrelic/newrelic.jar*
>>
>> Now, I have specified the absolute path of the NewRelic jar. There are 2
>> challenges that I see with this approach:
>>
>> 1. The NewRelic jar and config would have to be present on all the Yarn
>> slave nodes since that container can start at any of the Yarn nodes.
>> 2. A single Yarn cluster can be used to run multiple Samza applications,
>> so I would have to keep configuration for each application at each of the
>> Yarn slave nodes.
>>
>> In order to fix this, I am bundling NewRelic inside my tar file:
>>
>> // download new relic dependencies
>> task downloadNewrelic(type: Download) {
>> mkdir "newrelic"
>> src "
>> https://download.newrelic.com/newrelic/java-agent/newrelic-agent/current/newrelic-java.zip
>> "
>> dest file("newrelic")
>> }
>>
>> task buildTar(dependsOn: [build, downloadNewrelic], type: Tar) {
>> // unzip new relic dependencies and add to the archive
>> into(".") {
>> from zipTree(file("newrelic/newrelic-java.zip"))
>> }
>> }
>>
>> With this build configuration, I have an additional *newrelic* directory
>> inside my archive. Here is the directory structure:
>> my-job-name.tar.gz
>> bin
>> config
>> lib
>> newrelic
>> newrelic.jar
>> newrelic.yml
>> ...
>>
>> What I would like to achieve is to:
>> - Remove hardcoding the *javaagent* inside *task.opts* in the container
>> config.
>> - Specify the path of NewRelic agent in *task.opts* relative to where
>> the container files are present on the Yarn slave node.
>>
>> I hope I was clearer this time. Please let me know if any other
>> clarification is needed.
>>
>> Thanks,
>> Vaibhav Garg
>> +91-9505020924
>> vaibhavgarg90@gmail.com
>> LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>>
>>
>> On Sat, Feb 29, 2020 at 5:22 AM Yi Pan <ni...@gmail.com> wrote:
>>
>>> Hi, Vaibhav,
>>>
>>> Not quite sure whether I understand your ask. How do you package and
>>> distribute NewRelic jar in the YARN cluster? It would be much helpful if
>>> you can share your job configure that worked and the places that you
>>> would
>>> like to change.
>>>
>>> Best,
>>>
>>> -Yi
>>>
>>> On Fri, Feb 28, 2020 at 11:55 AM Vaibhav Garg <va...@gmail.com>
>>> wrote:
>>>
>>> > OK. I have been able to use NewRelic by specifying -javaagent option in
>>> > task.opts.
>>> >
>>> > However, I had to give the absolute path to the NewRelic jar file in
>>> order
>>> > to make it work.
>>> >
>>> > How can I give a path that is relative to the container since I would
>>> like
>>> > to integrate NewRelic dependencies along with other container files?
>>> >
>>> > Thanks,
>>> > Vaibhav Garg
>>> > +91-9505020924
>>> > vaibhavgarg90@gmail.com
>>> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>>> >
>>> >
>>> > On Sat, Feb 29, 2020 at 12:41 AM Vaibhav Garg <vaibhavgarg90@gmail.com
>>> >
>>> > wrote:
>>> >
>>> > > Yes. That's exactly what I am looking for.
>>> > >
>>> > > Let me try it out and I will get back to you if I face any issues.
>>> > >
>>> > > Thanks again
>>> > >
>>> > > Vaibhav Garg
>>> > > +91-9505020924
>>> > > vaibhavgarg90@gmail.com
>>> > > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>>> > >
>>> > >
>>> > > On Fri, Feb 28, 2020 at 1:23 PM Yi Pan <ni...@gmail.com> wrote:
>>> > >
>>> > >> Hi, Vaibhav,
>>> > >>
>>> > >> Check the description of task.opts in the configuration doc here:
>>> > >>
>>> > >>
>>> >
>>> http://samza.apache.org/learn/documentation/latest/jobs/samza-configurations.html
>>> > >>
>>> > >> Is this what you are looking for?
>>> > >>
>>> > >> On Thu, Feb 27, 2020 at 9:54 PM Vaibhav Garg <
>>> vaibhavgarg90@gmail.com>
>>> > >> wrote:
>>> > >>
>>> > >> > Hi,
>>> > >> >
>>> > >> > Any help here would be great.
>>> > >> >
>>> > >> > Thanks,
>>> > >> > Vaibhav Garg
>>> > >> > +91-9505020924
>>> > >> > vaibhavgarg90@gmail.com
>>> > >> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>>> > >> >
>>> > >> >
>>> > >> > On Thu, Feb 27, 2020 at 10:20 AM Vaibhav Garg <
>>> > vaibhavgarg90@gmail.com>
>>> > >> > wrote:
>>> > >> >
>>> > >> > > Dear Bharath,
>>> > >> > >
>>> > >> > > Thanks for your reply. I now realize that I put a very ambiguous
>>> > >> question
>>> > >> > > to the community. Here is another attempt:
>>> > >> > >
>>> > >> > > I have set up a Yarn cluster that is configured to run
>>> containers of
>>> > >> > > multiple Samza jobs.
>>> > >> > >
>>> > >> > > I would like to treat all the containers of the job as a single
>>> > >> > > application in New Relic (as expected).
>>> > >> > >
>>> > >> > > Since the containers of the application can be killed on one
>>> Yarn
>>> > node
>>> > >> > and
>>> > >> > > can start on another Yarn node, I would have to specify New
>>> Relic
>>> > >> > > environment settings at the container start.
>>> > >> > >
>>> > >> > > Now, I am not sure how to specify java agent (to include New
>>> Relic
>>> > >> java
>>> > >> > > agent) and pass additional arguments such as New Relic
>>> environment
>>> > >> every
>>> > >> > > time the container starts.
>>> > >> > >
>>> > >> > > Please let me know if my understanding is wrong or some other
>>> > >> > > clarification is needed.
>>> > >> > >
>>> > >> > > Thanks in advance,
>>> > >> > > Vaibhav Garg
>>> > >> > > +91-9505020924
>>> > >> > > vaibhavgarg90@gmail.com
>>> > >> > > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>>> > >> > >
>>> > >> > >
>>> > >> > > On Thu, Feb 27, 2020 at 8:25 AM Bharath Kumara Subramanian <
>>> > >> > > codin.martial@gmail.com> wrote:
>>> > >> > >
>>> > >> > >> Hi,
>>> > >> > >>
>>> > >> > >> I am not sure I fully understand the ask.
>>> > >> > >>
>>> > >> > >> IIUC, Samza doesn't have native integration with New relic.
>>> > However,
>>> > >> you
>>> > >> > >> should still be able to integrate your application with New
>>> relic
>>> > on
>>> > >> > your
>>> > >> > >> end without native support.
>>> > >> > >> If you are particularly looking to integrate native Samza
>>> metrics
>>> > w/
>>> > >> New
>>> > >> > >> relic, you might need to implement your own custom metrics
>>> > reporter.
>>> > >> You
>>> > >> > >> can find more details here
>>> > >> > >> <
>>> > >> > >>
>>> > >> >
>>> > >>
>>> >
>>> https://samza.apache.org/learn/documentation/latest/operations/monitoring.html#customreporter
>>> > >> > >> >
>>> > >> > >>
>>> > >> > >> Thanks,
>>> > >> > >> Bharath
>>> > >> > >>
>>> > >> > >> On Wed, Feb 26, 2020 at 3:15 AM Vaibhav Garg <
>>> > >> vaibhavgarg90@gmail.com>
>>> > >> > >> wrote:
>>> > >> > >>
>>> > >> > >> > Hi,
>>> > >> > >> >
>>> > >> > >> > I want to integrate New Relic in my Samza jobs. Can anyone
>>> help
>>> > >> with
>>> > >> > >> this,
>>> > >> > >> > please?
>>> > >> > >> >
>>> > >> > >> > Thanks,
>>> > >> > >> > Vaibhav Garg
>>> > >> > >> > +91-9505020924
>>> > >> > >> > vaibhavgarg90@gmail.com
>>> > >> > >> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>>> > >> > >> >
>>> > >> > >>
>>> > >> > >
>>> > >> >
>>> > >>
>>> > >
>>> >
>>>
>>
Re: Samza - New Relic integration
Posted by Jagadish Venkatraman <ja...@gmail.com>.
That's great to hear Vaibhav. Feel free to reach out to the community if
you have further questions!
Best
Jagadish
On Sunday, March 1, 2020, Vaibhav Garg <va...@gmail.com> wrote:
> Ok. That worked for me.
>
> Thank you, Bharath, Yi Pan, and Jagadish for your help.
>
> Thanks,
> Vaibhav Garg
> +91-9505020924
> vaibhavgarg90@gmail.com
> LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>
>
> On Sun, Mar 1, 2020 at 12:56 PM Vaibhav Garg <va...@gmail.com>
> wrote:
>
> > Ok. I will try that.
> >
> > Thanks,
> > Vaibhav Garg
> > +91-9505020924
> > vaibhavgarg90@gmail.com
> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> >
> >
> > On Sun, Mar 1, 2020 at 12:31 PM Jagadish Venkatraman <
> > jagadish1989@gmail.com> wrote:
> >
> >> Hi Vaibhav,
> >>
> >> Can you try ./__package/newrelic/newrelic.jar?
> >>
> >> If that doesn't work, I'd recommend doing the following:
> >> 1. Print the current working directory of your Samza container JVM
> >> 2. Use (1) to determine a relative reference to the directory path
> within
> >> your package
> >>
> >> Cheers
> >> Jagadish
> >>
> >> On Saturday, February 29, 2020, Vaibhav Garg <va...@gmail.com>
> >> wrote:
> >>
> >> > I would be really grateful if someone could help me with this.
> >> >
> >> > Thanks,
> >> > Vaibhav Garg
> >> > +91-9505020924
> >> > vaibhavgarg90@gmail.com
> >> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> >> >
> >> >
> >> > On Sat, Feb 29, 2020 at 9:02 PM Vaibhav Garg <vaibhavgarg90@gmail.com
> >
> >> > wrote:
> >> >
> >> > > Please do let me know if some other clarification is needed.
> >> > >
> >> > > Thanks,
> >> > > Vaibhav Garg
> >> > > +91-9505020924
> >> > > vaibhavgarg90@gmail.com
> >> > > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> >> > >
> >> > >
> >> > > On Sat, Feb 29, 2020 at 10:31 AM Vaibhav Garg <
> >> vaibhavgarg90@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> OK. Here is what I have inside my configuration in order to start
> >> > >> NewRelic at the time of the container start:
> >> > >>
> >> > >> task.opts=-Xmx2048m -XX:+UseConcMarkSweepGC
> >> > >> *-javaagent:/usr/home/vaibhav/newrelic/newrelic.jar*
> >> > >>
> >> > >> Now, I have specified the absolute path of the NewRelic jar. There
> >> are 2
> >> > >> challenges that I see with this approach:
> >> > >>
> >> > >> 1. The NewRelic jar and config would have to be present on all the
> >> Yarn
> >> > >> slave nodes since that container can start at any of the Yarn
> nodes.
> >> > >> 2. A single Yarn cluster can be used to run multiple Samza
> >> applications,
> >> > >> so I would have to keep configuration for each application at each
> of
> >> > the
> >> > >> Yarn slave nodes.
> >> > >>
> >> > >> In order to fix this, I am bundling NewRelic inside my tar file:
> >> > >>
> >> > >> // download new relic dependencies
> >> > >> task downloadNewrelic(type: Download) {
> >> > >> mkdir "newrelic"
> >> > >> src "
> >> > >> https://download.newrelic.com/newrelic/java-agent/newrelic-
> >> > agent/current/newrelic-java.zip
> >> > >> "
> >> > >> dest file("newrelic")
> >> > >> }
> >> > >>
> >> > >> task buildTar(dependsOn: [build, downloadNewrelic], type: Tar) {
> >> > >> // unzip new relic dependencies and add to the archive
> >> > >> into(".") {
> >> > >> from zipTree(file("newrelic/newrelic-java.zip"))
> >> > >> }
> >> > >> }
> >> > >>
> >> > >> With this build configuration, I have an additional *newrelic*
> >> directory
> >> > >> inside my archive. Here is the directory structure:
> >> > >> my-job-name.tar.gz
> >> > >> bin
> >> > >> config
> >> > >> lib
> >> > >> newrelic
> >> > >> newrelic.jar
> >> > >> newrelic.yml
> >> > >> ...
> >> > >>
> >> > >> What I would like to achieve is to:
> >> > >> - Remove hardcoding the *javaagent* inside *task.opts* in the
> >> container
> >> > >> config.
> >> > >> - Specify the path of NewRelic agent in *task.opts* relative to
> where
> >> > >> the container files are present on the Yarn slave node.
> >> > >>
> >> > >> I hope I was clearer this time. Please let me know if any other
> >> > >> clarification is needed.
> >> > >>
> >> > >> Thanks,
> >> > >> Vaibhav Garg
> >> > >> +91-9505020924
> >> > >> vaibhavgarg90@gmail.com
> >> > >> LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> >> > >>
> >> > >>
> >> > >> On Sat, Feb 29, 2020 at 5:22 AM Yi Pan <ni...@gmail.com>
> wrote:
> >> > >>
> >> > >>> Hi, Vaibhav,
> >> > >>>
> >> > >>> Not quite sure whether I understand your ask. How do you package
> and
> >> > >>> distribute NewRelic jar in the YARN cluster? It would be much
> >> helpful
> >> > if
> >> > >>> you can share your job configure that worked and the places that
> you
> >> > >>> would
> >> > >>> like to change.
> >> > >>>
> >> > >>> Best,
> >> > >>>
> >> > >>> -Yi
> >> > >>>
> >> > >>> On Fri, Feb 28, 2020 at 11:55 AM Vaibhav Garg <
> >> vaibhavgarg90@gmail.com
> >> > >
> >> > >>> wrote:
> >> > >>>
> >> > >>> > OK. I have been able to use NewRelic by specifying -javaagent
> >> option
> >> > in
> >> > >>> > task.opts.
> >> > >>> >
> >> > >>> > However, I had to give the absolute path to the NewRelic jar
> file
> >> in
> >> > >>> order
> >> > >>> > to make it work.
> >> > >>> >
> >> > >>> > How can I give a path that is relative to the container since I
> >> would
> >> > >>> like
> >> > >>> > to integrate NewRelic dependencies along with other container
> >> files?
> >> > >>> >
> >> > >>> > Thanks,
> >> > >>> > Vaibhav Garg
> >> > >>> > +91-9505020924
> >> > >>> > vaibhavgarg90@gmail.com
> >> > >>> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> >> > >>> >
> >> > >>> >
> >> > >>> > On Sat, Feb 29, 2020 at 12:41 AM Vaibhav Garg <
> >> > vaibhavgarg90@gmail.com
> >> > >>> >
> >> > >>> > wrote:
> >> > >>> >
> >> > >>> > > Yes. That's exactly what I am looking for.
> >> > >>> > >
> >> > >>> > > Let me try it out and I will get back to you if I face any
> >> issues.
> >> > >>> > >
> >> > >>> > > Thanks again
> >> > >>> > >
> >> > >>> > > Vaibhav Garg
> >> > >>> > > +91-9505020924
> >> > >>> > > vaibhavgarg90@gmail.com
> >> > >>> > > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> >> > >>> > >
> >> > >>> > >
> >> > >>> > > On Fri, Feb 28, 2020 at 1:23 PM Yi Pan <ni...@gmail.com>
> >> > wrote:
> >> > >>> > >
> >> > >>> > >> Hi, Vaibhav,
> >> > >>> > >>
> >> > >>> > >> Check the description of task.opts in the configuration doc
> >> here:
> >> > >>> > >>
> >> > >>> > >>
> >> > >>> >
> >> > >>> http://samza.apache.org/learn/documentation/latest/jobs/
> >> > samza-configurations.html
> >> > >>> > >>
> >> > >>> > >> Is this what you are looking for?
> >> > >>> > >>
> >> > >>> > >> On Thu, Feb 27, 2020 at 9:54 PM Vaibhav Garg <
> >> > >>> vaibhavgarg90@gmail.com>
> >> > >>> > >> wrote:
> >> > >>> > >>
> >> > >>> > >> > Hi,
> >> > >>> > >> >
> >> > >>> > >> > Any help here would be great.
> >> > >>> > >> >
> >> > >>> > >> > Thanks,
> >> > >>> > >> > Vaibhav Garg
> >> > >>> > >> > +91-9505020924
> >> > >>> > >> > vaibhavgarg90@gmail.com
> >> > >>> > >> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> >> > >>> > >> >
> >> > >>> > >> >
> >> > >>> > >> > On Thu, Feb 27, 2020 at 10:20 AM Vaibhav Garg <
> >> > >>> > vaibhavgarg90@gmail.com>
> >> > >>> > >> > wrote:
> >> > >>> > >> >
> >> > >>> > >> > > Dear Bharath,
> >> > >>> > >> > >
> >> > >>> > >> > > Thanks for your reply. I now realize that I put a very
> >> > ambiguous
> >> > >>> > >> question
> >> > >>> > >> > > to the community. Here is another attempt:
> >> > >>> > >> > >
> >> > >>> > >> > > I have set up a Yarn cluster that is configured to run
> >> > >>> containers of
> >> > >>> > >> > > multiple Samza jobs.
> >> > >>> > >> > >
> >> > >>> > >> > > I would like to treat all the containers of the job as a
> >> > single
> >> > >>> > >> > > application in New Relic (as expected).
> >> > >>> > >> > >
> >> > >>> > >> > > Since the containers of the application can be killed on
> >> one
> >> > >>> Yarn
> >> > >>> > node
> >> > >>> > >> > and
> >> > >>> > >> > > can start on another Yarn node, I would have to specify
> New
> >> > >>> Relic
> >> > >>> > >> > > environment settings at the container start.
> >> > >>> > >> > >
> >> > >>> > >> > > Now, I am not sure how to specify java agent (to include
> >> New
> >> > >>> Relic
> >> > >>> > >> java
> >> > >>> > >> > > agent) and pass additional arguments such as New Relic
> >> > >>> environment
> >> > >>> > >> every
> >> > >>> > >> > > time the container starts.
> >> > >>> > >> > >
> >> > >>> > >> > > Please let me know if my understanding is wrong or some
> >> other
> >> > >>> > >> > > clarification is needed.
> >> > >>> > >> > >
> >> > >>> > >> > > Thanks in advance,
> >> > >>> > >> > > Vaibhav Garg
> >> > >>> > >> > > +91-9505020924
> >> > >>> > >> > > vaibhavgarg90@gmail.com
> >> > >>> > >> > > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> >> > >>> > >> > >
> >> > >>> > >> > >
> >> > >>> > >> > > On Thu, Feb 27, 2020 at 8:25 AM Bharath Kumara
> Subramanian
> >> <
> >> > >>> > >> > > codin.martial@gmail.com> wrote:
> >> > >>> > >> > >
> >> > >>> > >> > >> Hi,
> >> > >>> > >> > >>
> >> > >>> > >> > >> I am not sure I fully understand the ask.
> >> > >>> > >> > >>
> >> > >>> > >> > >> IIUC, Samza doesn't have native integration with New
> >> relic.
> >> > >>> > However,
> >> > >>> > >> you
> >> > >>> > >> > >> should still be able to integrate your application with
> >> New
> >> > >>> relic
> >> > >>> > on
> >> > >>> > >> > your
> >> > >>> > >> > >> end without native support.
> >> > >>> > >> > >> If you are particularly looking to integrate native
> Samza
> >> > >>> metrics
> >> > >>> > w/
> >> > >>> > >> New
> >> > >>> > >> > >> relic, you might need to implement your own custom
> metrics
> >> > >>> > reporter.
> >> > >>> > >> You
> >> > >>> > >> > >> can find more details here
> >> > >>> > >> > >> <
> >> > >>> > >> > >>
> >> > >>> > >> >
> >> > >>> > >>
> >> > >>> >
> >> > >>> https://samza.apache.org/learn/documentation/latest/
> >> > operations/monitoring.html#customreporter
> >> > >>> > >> > >> >
> >> > >>> > >> > >>
> >> > >>> > >> > >> Thanks,
> >> > >>> > >> > >> Bharath
> >> > >>> > >> > >>
> >> > >>> > >> > >> On Wed, Feb 26, 2020 at 3:15 AM Vaibhav Garg <
> >> > >>> > >> vaibhavgarg90@gmail.com>
> >> > >>> > >> > >> wrote:
> >> > >>> > >> > >>
> >> > >>> > >> > >> > Hi,
> >> > >>> > >> > >> >
> >> > >>> > >> > >> > I want to integrate New Relic in my Samza jobs. Can
> >> anyone
> >> > >>> help
> >> > >>> > >> with
> >> > >>> > >> > >> this,
> >> > >>> > >> > >> > please?
> >> > >>> > >> > >> >
> >> > >>> > >> > >> > Thanks,
> >> > >>> > >> > >> > Vaibhav Garg
> >> > >>> > >> > >> > +91-9505020924
> >> > >>> > >> > >> > vaibhavgarg90@gmail.com
> >> > >>> > >> > >> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> >> > >>> > >> > >> >
> >> > >>> > >> > >>
> >> > >>> > >> > >
> >> > >>> > >> >
> >> > >>> > >>
> >> > >>> > >
> >> > >>> >
> >> > >>>
> >> > >>
> >> >
> >>
> >>
> >> --
> >> Jagadish
> >>
> >
>
--
Jagadish
Re: Samza - New Relic integration
Posted by Vaibhav Garg <va...@gmail.com>.
Ok. That worked for me.
Thank you, Bharath, Yi Pan, and Jagadish for your help.
Thanks,
Vaibhav Garg
+91-9505020924
vaibhavgarg90@gmail.com
LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
On Sun, Mar 1, 2020 at 12:56 PM Vaibhav Garg <va...@gmail.com>
wrote:
> Ok. I will try that.
>
> Thanks,
> Vaibhav Garg
> +91-9505020924
> vaibhavgarg90@gmail.com
> LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>
>
> On Sun, Mar 1, 2020 at 12:31 PM Jagadish Venkatraman <
> jagadish1989@gmail.com> wrote:
>
>> Hi Vaibhav,
>>
>> Can you try ./__package/newrelic/newrelic.jar?
>>
>> If that doesn't work, I'd recommend doing the following:
>> 1. Print the current working directory of your Samza container JVM
>> 2. Use (1) to determine a relative reference to the directory path within
>> your package
>>
>> Cheers
>> Jagadish
>>
>> On Saturday, February 29, 2020, Vaibhav Garg <va...@gmail.com>
>> wrote:
>>
>> > I would be really grateful if someone could help me with this.
>> >
>> > Thanks,
>> > Vaibhav Garg
>> > +91-9505020924
>> > vaibhavgarg90@gmail.com
>> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>> >
>> >
>> > On Sat, Feb 29, 2020 at 9:02 PM Vaibhav Garg <va...@gmail.com>
>> > wrote:
>> >
>> > > Please do let me know if some other clarification is needed.
>> > >
>> > > Thanks,
>> > > Vaibhav Garg
>> > > +91-9505020924
>> > > vaibhavgarg90@gmail.com
>> > > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>> > >
>> > >
>> > > On Sat, Feb 29, 2020 at 10:31 AM Vaibhav Garg <
>> vaibhavgarg90@gmail.com>
>> > > wrote:
>> > >
>> > >> OK. Here is what I have inside my configuration in order to start
>> > >> NewRelic at the time of the container start:
>> > >>
>> > >> task.opts=-Xmx2048m -XX:+UseConcMarkSweepGC
>> > >> *-javaagent:/usr/home/vaibhav/newrelic/newrelic.jar*
>> > >>
>> > >> Now, I have specified the absolute path of the NewRelic jar. There
>> are 2
>> > >> challenges that I see with this approach:
>> > >>
>> > >> 1. The NewRelic jar and config would have to be present on all the
>> Yarn
>> > >> slave nodes since that container can start at any of the Yarn nodes.
>> > >> 2. A single Yarn cluster can be used to run multiple Samza
>> applications,
>> > >> so I would have to keep configuration for each application at each of
>> > the
>> > >> Yarn slave nodes.
>> > >>
>> > >> In order to fix this, I am bundling NewRelic inside my tar file:
>> > >>
>> > >> // download new relic dependencies
>> > >> task downloadNewrelic(type: Download) {
>> > >> mkdir "newrelic"
>> > >> src "
>> > >> https://download.newrelic.com/newrelic/java-agent/newrelic-
>> > agent/current/newrelic-java.zip
>> > >> "
>> > >> dest file("newrelic")
>> > >> }
>> > >>
>> > >> task buildTar(dependsOn: [build, downloadNewrelic], type: Tar) {
>> > >> // unzip new relic dependencies and add to the archive
>> > >> into(".") {
>> > >> from zipTree(file("newrelic/newrelic-java.zip"))
>> > >> }
>> > >> }
>> > >>
>> > >> With this build configuration, I have an additional *newrelic*
>> directory
>> > >> inside my archive. Here is the directory structure:
>> > >> my-job-name.tar.gz
>> > >> bin
>> > >> config
>> > >> lib
>> > >> newrelic
>> > >> newrelic.jar
>> > >> newrelic.yml
>> > >> ...
>> > >>
>> > >> What I would like to achieve is to:
>> > >> - Remove hardcoding the *javaagent* inside *task.opts* in the
>> container
>> > >> config.
>> > >> - Specify the path of NewRelic agent in *task.opts* relative to where
>> > >> the container files are present on the Yarn slave node.
>> > >>
>> > >> I hope I was clearer this time. Please let me know if any other
>> > >> clarification is needed.
>> > >>
>> > >> Thanks,
>> > >> Vaibhav Garg
>> > >> +91-9505020924
>> > >> vaibhavgarg90@gmail.com
>> > >> LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>> > >>
>> > >>
>> > >> On Sat, Feb 29, 2020 at 5:22 AM Yi Pan <ni...@gmail.com> wrote:
>> > >>
>> > >>> Hi, Vaibhav,
>> > >>>
>> > >>> Not quite sure whether I understand your ask. How do you package and
>> > >>> distribute NewRelic jar in the YARN cluster? It would be much
>> helpful
>> > if
>> > >>> you can share your job configure that worked and the places that you
>> > >>> would
>> > >>> like to change.
>> > >>>
>> > >>> Best,
>> > >>>
>> > >>> -Yi
>> > >>>
>> > >>> On Fri, Feb 28, 2020 at 11:55 AM Vaibhav Garg <
>> vaibhavgarg90@gmail.com
>> > >
>> > >>> wrote:
>> > >>>
>> > >>> > OK. I have been able to use NewRelic by specifying -javaagent
>> option
>> > in
>> > >>> > task.opts.
>> > >>> >
>> > >>> > However, I had to give the absolute path to the NewRelic jar file
>> in
>> > >>> order
>> > >>> > to make it work.
>> > >>> >
>> > >>> > How can I give a path that is relative to the container since I
>> would
>> > >>> like
>> > >>> > to integrate NewRelic dependencies along with other container
>> files?
>> > >>> >
>> > >>> > Thanks,
>> > >>> > Vaibhav Garg
>> > >>> > +91-9505020924
>> > >>> > vaibhavgarg90@gmail.com
>> > >>> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>> > >>> >
>> > >>> >
>> > >>> > On Sat, Feb 29, 2020 at 12:41 AM Vaibhav Garg <
>> > vaibhavgarg90@gmail.com
>> > >>> >
>> > >>> > wrote:
>> > >>> >
>> > >>> > > Yes. That's exactly what I am looking for.
>> > >>> > >
>> > >>> > > Let me try it out and I will get back to you if I face any
>> issues.
>> > >>> > >
>> > >>> > > Thanks again
>> > >>> > >
>> > >>> > > Vaibhav Garg
>> > >>> > > +91-9505020924
>> > >>> > > vaibhavgarg90@gmail.com
>> > >>> > > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>> > >>> > >
>> > >>> > >
>> > >>> > > On Fri, Feb 28, 2020 at 1:23 PM Yi Pan <ni...@gmail.com>
>> > wrote:
>> > >>> > >
>> > >>> > >> Hi, Vaibhav,
>> > >>> > >>
>> > >>> > >> Check the description of task.opts in the configuration doc
>> here:
>> > >>> > >>
>> > >>> > >>
>> > >>> >
>> > >>> http://samza.apache.org/learn/documentation/latest/jobs/
>> > samza-configurations.html
>> > >>> > >>
>> > >>> > >> Is this what you are looking for?
>> > >>> > >>
>> > >>> > >> On Thu, Feb 27, 2020 at 9:54 PM Vaibhav Garg <
>> > >>> vaibhavgarg90@gmail.com>
>> > >>> > >> wrote:
>> > >>> > >>
>> > >>> > >> > Hi,
>> > >>> > >> >
>> > >>> > >> > Any help here would be great.
>> > >>> > >> >
>> > >>> > >> > Thanks,
>> > >>> > >> > Vaibhav Garg
>> > >>> > >> > +91-9505020924
>> > >>> > >> > vaibhavgarg90@gmail.com
>> > >>> > >> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>> > >>> > >> >
>> > >>> > >> >
>> > >>> > >> > On Thu, Feb 27, 2020 at 10:20 AM Vaibhav Garg <
>> > >>> > vaibhavgarg90@gmail.com>
>> > >>> > >> > wrote:
>> > >>> > >> >
>> > >>> > >> > > Dear Bharath,
>> > >>> > >> > >
>> > >>> > >> > > Thanks for your reply. I now realize that I put a very
>> > ambiguous
>> > >>> > >> question
>> > >>> > >> > > to the community. Here is another attempt:
>> > >>> > >> > >
>> > >>> > >> > > I have set up a Yarn cluster that is configured to run
>> > >>> containers of
>> > >>> > >> > > multiple Samza jobs.
>> > >>> > >> > >
>> > >>> > >> > > I would like to treat all the containers of the job as a
>> > single
>> > >>> > >> > > application in New Relic (as expected).
>> > >>> > >> > >
>> > >>> > >> > > Since the containers of the application can be killed on
>> one
>> > >>> Yarn
>> > >>> > node
>> > >>> > >> > and
>> > >>> > >> > > can start on another Yarn node, I would have to specify New
>> > >>> Relic
>> > >>> > >> > > environment settings at the container start.
>> > >>> > >> > >
>> > >>> > >> > > Now, I am not sure how to specify java agent (to include
>> New
>> > >>> Relic
>> > >>> > >> java
>> > >>> > >> > > agent) and pass additional arguments such as New Relic
>> > >>> environment
>> > >>> > >> every
>> > >>> > >> > > time the container starts.
>> > >>> > >> > >
>> > >>> > >> > > Please let me know if my understanding is wrong or some
>> other
>> > >>> > >> > > clarification is needed.
>> > >>> > >> > >
>> > >>> > >> > > Thanks in advance,
>> > >>> > >> > > Vaibhav Garg
>> > >>> > >> > > +91-9505020924
>> > >>> > >> > > vaibhavgarg90@gmail.com
>> > >>> > >> > > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>> > >>> > >> > >
>> > >>> > >> > >
>> > >>> > >> > > On Thu, Feb 27, 2020 at 8:25 AM Bharath Kumara Subramanian
>> <
>> > >>> > >> > > codin.martial@gmail.com> wrote:
>> > >>> > >> > >
>> > >>> > >> > >> Hi,
>> > >>> > >> > >>
>> > >>> > >> > >> I am not sure I fully understand the ask.
>> > >>> > >> > >>
>> > >>> > >> > >> IIUC, Samza doesn't have native integration with New
>> relic.
>> > >>> > However,
>> > >>> > >> you
>> > >>> > >> > >> should still be able to integrate your application with
>> New
>> > >>> relic
>> > >>> > on
>> > >>> > >> > your
>> > >>> > >> > >> end without native support.
>> > >>> > >> > >> If you are particularly looking to integrate native Samza
>> > >>> metrics
>> > >>> > w/
>> > >>> > >> New
>> > >>> > >> > >> relic, you might need to implement your own custom metrics
>> > >>> > reporter.
>> > >>> > >> You
>> > >>> > >> > >> can find more details here
>> > >>> > >> > >> <
>> > >>> > >> > >>
>> > >>> > >> >
>> > >>> > >>
>> > >>> >
>> > >>> https://samza.apache.org/learn/documentation/latest/
>> > operations/monitoring.html#customreporter
>> > >>> > >> > >> >
>> > >>> > >> > >>
>> > >>> > >> > >> Thanks,
>> > >>> > >> > >> Bharath
>> > >>> > >> > >>
>> > >>> > >> > >> On Wed, Feb 26, 2020 at 3:15 AM Vaibhav Garg <
>> > >>> > >> vaibhavgarg90@gmail.com>
>> > >>> > >> > >> wrote:
>> > >>> > >> > >>
>> > >>> > >> > >> > Hi,
>> > >>> > >> > >> >
>> > >>> > >> > >> > I want to integrate New Relic in my Samza jobs. Can
>> anyone
>> > >>> help
>> > >>> > >> with
>> > >>> > >> > >> this,
>> > >>> > >> > >> > please?
>> > >>> > >> > >> >
>> > >>> > >> > >> > Thanks,
>> > >>> > >> > >> > Vaibhav Garg
>> > >>> > >> > >> > +91-9505020924
>> > >>> > >> > >> > vaibhavgarg90@gmail.com
>> > >>> > >> > >> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>> > >>> > >> > >> >
>> > >>> > >> > >>
>> > >>> > >> > >
>> > >>> > >> >
>> > >>> > >>
>> > >>> > >
>> > >>> >
>> > >>>
>> > >>
>> >
>>
>>
>> --
>> Jagadish
>>
>
Re: Samza - New Relic integration
Posted by Vaibhav Garg <va...@gmail.com>.
Ok. I will try that.
Thanks,
Vaibhav Garg
+91-9505020924
vaibhavgarg90@gmail.com
LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
On Sun, Mar 1, 2020 at 12:31 PM Jagadish Venkatraman <ja...@gmail.com>
wrote:
> Hi Vaibhav,
>
> Can you try ./__package/newrelic/newrelic.jar?
>
> If that doesn't work, I'd recommend doing the following:
> 1. Print the current working directory of your Samza container JVM
> 2. Use (1) to determine a relative reference to the directory path within
> your package
>
> Cheers
> Jagadish
>
> On Saturday, February 29, 2020, Vaibhav Garg <va...@gmail.com>
> wrote:
>
> > I would be really grateful if someone could help me with this.
> >
> > Thanks,
> > Vaibhav Garg
> > +91-9505020924
> > vaibhavgarg90@gmail.com
> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> >
> >
> > On Sat, Feb 29, 2020 at 9:02 PM Vaibhav Garg <va...@gmail.com>
> > wrote:
> >
> > > Please do let me know if some other clarification is needed.
> > >
> > > Thanks,
> > > Vaibhav Garg
> > > +91-9505020924
> > > vaibhavgarg90@gmail.com
> > > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> > >
> > >
> > > On Sat, Feb 29, 2020 at 10:31 AM Vaibhav Garg <vaibhavgarg90@gmail.com
> >
> > > wrote:
> > >
> > >> OK. Here is what I have inside my configuration in order to start
> > >> NewRelic at the time of the container start:
> > >>
> > >> task.opts=-Xmx2048m -XX:+UseConcMarkSweepGC
> > >> *-javaagent:/usr/home/vaibhav/newrelic/newrelic.jar*
> > >>
> > >> Now, I have specified the absolute path of the NewRelic jar. There
> are 2
> > >> challenges that I see with this approach:
> > >>
> > >> 1. The NewRelic jar and config would have to be present on all the
> Yarn
> > >> slave nodes since that container can start at any of the Yarn nodes.
> > >> 2. A single Yarn cluster can be used to run multiple Samza
> applications,
> > >> so I would have to keep configuration for each application at each of
> > the
> > >> Yarn slave nodes.
> > >>
> > >> In order to fix this, I am bundling NewRelic inside my tar file:
> > >>
> > >> // download new relic dependencies
> > >> task downloadNewrelic(type: Download) {
> > >> mkdir "newrelic"
> > >> src "
> > >> https://download.newrelic.com/newrelic/java-agent/newrelic-
> > agent/current/newrelic-java.zip
> > >> "
> > >> dest file("newrelic")
> > >> }
> > >>
> > >> task buildTar(dependsOn: [build, downloadNewrelic], type: Tar) {
> > >> // unzip new relic dependencies and add to the archive
> > >> into(".") {
> > >> from zipTree(file("newrelic/newrelic-java.zip"))
> > >> }
> > >> }
> > >>
> > >> With this build configuration, I have an additional *newrelic*
> directory
> > >> inside my archive. Here is the directory structure:
> > >> my-job-name.tar.gz
> > >> bin
> > >> config
> > >> lib
> > >> newrelic
> > >> newrelic.jar
> > >> newrelic.yml
> > >> ...
> > >>
> > >> What I would like to achieve is to:
> > >> - Remove hardcoding the *javaagent* inside *task.opts* in the
> container
> > >> config.
> > >> - Specify the path of NewRelic agent in *task.opts* relative to where
> > >> the container files are present on the Yarn slave node.
> > >>
> > >> I hope I was clearer this time. Please let me know if any other
> > >> clarification is needed.
> > >>
> > >> Thanks,
> > >> Vaibhav Garg
> > >> +91-9505020924
> > >> vaibhavgarg90@gmail.com
> > >> LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> > >>
> > >>
> > >> On Sat, Feb 29, 2020 at 5:22 AM Yi Pan <ni...@gmail.com> wrote:
> > >>
> > >>> Hi, Vaibhav,
> > >>>
> > >>> Not quite sure whether I understand your ask. How do you package and
> > >>> distribute NewRelic jar in the YARN cluster? It would be much helpful
> > if
> > >>> you can share your job configure that worked and the places that you
> > >>> would
> > >>> like to change.
> > >>>
> > >>> Best,
> > >>>
> > >>> -Yi
> > >>>
> > >>> On Fri, Feb 28, 2020 at 11:55 AM Vaibhav Garg <
> vaibhavgarg90@gmail.com
> > >
> > >>> wrote:
> > >>>
> > >>> > OK. I have been able to use NewRelic by specifying -javaagent
> option
> > in
> > >>> > task.opts.
> > >>> >
> > >>> > However, I had to give the absolute path to the NewRelic jar file
> in
> > >>> order
> > >>> > to make it work.
> > >>> >
> > >>> > How can I give a path that is relative to the container since I
> would
> > >>> like
> > >>> > to integrate NewRelic dependencies along with other container
> files?
> > >>> >
> > >>> > Thanks,
> > >>> > Vaibhav Garg
> > >>> > +91-9505020924
> > >>> > vaibhavgarg90@gmail.com
> > >>> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> > >>> >
> > >>> >
> > >>> > On Sat, Feb 29, 2020 at 12:41 AM Vaibhav Garg <
> > vaibhavgarg90@gmail.com
> > >>> >
> > >>> > wrote:
> > >>> >
> > >>> > > Yes. That's exactly what I am looking for.
> > >>> > >
> > >>> > > Let me try it out and I will get back to you if I face any
> issues.
> > >>> > >
> > >>> > > Thanks again
> > >>> > >
> > >>> > > Vaibhav Garg
> > >>> > > +91-9505020924
> > >>> > > vaibhavgarg90@gmail.com
> > >>> > > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> > >>> > >
> > >>> > >
> > >>> > > On Fri, Feb 28, 2020 at 1:23 PM Yi Pan <ni...@gmail.com>
> > wrote:
> > >>> > >
> > >>> > >> Hi, Vaibhav,
> > >>> > >>
> > >>> > >> Check the description of task.opts in the configuration doc
> here:
> > >>> > >>
> > >>> > >>
> > >>> >
> > >>> http://samza.apache.org/learn/documentation/latest/jobs/
> > samza-configurations.html
> > >>> > >>
> > >>> > >> Is this what you are looking for?
> > >>> > >>
> > >>> > >> On Thu, Feb 27, 2020 at 9:54 PM Vaibhav Garg <
> > >>> vaibhavgarg90@gmail.com>
> > >>> > >> wrote:
> > >>> > >>
> > >>> > >> > Hi,
> > >>> > >> >
> > >>> > >> > Any help here would be great.
> > >>> > >> >
> > >>> > >> > Thanks,
> > >>> > >> > Vaibhav Garg
> > >>> > >> > +91-9505020924
> > >>> > >> > vaibhavgarg90@gmail.com
> > >>> > >> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > On Thu, Feb 27, 2020 at 10:20 AM Vaibhav Garg <
> > >>> > vaibhavgarg90@gmail.com>
> > >>> > >> > wrote:
> > >>> > >> >
> > >>> > >> > > Dear Bharath,
> > >>> > >> > >
> > >>> > >> > > Thanks for your reply. I now realize that I put a very
> > ambiguous
> > >>> > >> question
> > >>> > >> > > to the community. Here is another attempt:
> > >>> > >> > >
> > >>> > >> > > I have set up a Yarn cluster that is configured to run
> > >>> containers of
> > >>> > >> > > multiple Samza jobs.
> > >>> > >> > >
> > >>> > >> > > I would like to treat all the containers of the job as a
> > single
> > >>> > >> > > application in New Relic (as expected).
> > >>> > >> > >
> > >>> > >> > > Since the containers of the application can be killed on one
> > >>> Yarn
> > >>> > node
> > >>> > >> > and
> > >>> > >> > > can start on another Yarn node, I would have to specify New
> > >>> Relic
> > >>> > >> > > environment settings at the container start.
> > >>> > >> > >
> > >>> > >> > > Now, I am not sure how to specify java agent (to include New
> > >>> Relic
> > >>> > >> java
> > >>> > >> > > agent) and pass additional arguments such as New Relic
> > >>> environment
> > >>> > >> every
> > >>> > >> > > time the container starts.
> > >>> > >> > >
> > >>> > >> > > Please let me know if my understanding is wrong or some
> other
> > >>> > >> > > clarification is needed.
> > >>> > >> > >
> > >>> > >> > > Thanks in advance,
> > >>> > >> > > Vaibhav Garg
> > >>> > >> > > +91-9505020924
> > >>> > >> > > vaibhavgarg90@gmail.com
> > >>> > >> > > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> > >>> > >> > >
> > >>> > >> > >
> > >>> > >> > > On Thu, Feb 27, 2020 at 8:25 AM Bharath Kumara Subramanian <
> > >>> > >> > > codin.martial@gmail.com> wrote:
> > >>> > >> > >
> > >>> > >> > >> Hi,
> > >>> > >> > >>
> > >>> > >> > >> I am not sure I fully understand the ask.
> > >>> > >> > >>
> > >>> > >> > >> IIUC, Samza doesn't have native integration with New relic.
> > >>> > However,
> > >>> > >> you
> > >>> > >> > >> should still be able to integrate your application with New
> > >>> relic
> > >>> > on
> > >>> > >> > your
> > >>> > >> > >> end without native support.
> > >>> > >> > >> If you are particularly looking to integrate native Samza
> > >>> metrics
> > >>> > w/
> > >>> > >> New
> > >>> > >> > >> relic, you might need to implement your own custom metrics
> > >>> > reporter.
> > >>> > >> You
> > >>> > >> > >> can find more details here
> > >>> > >> > >> <
> > >>> > >> > >>
> > >>> > >> >
> > >>> > >>
> > >>> >
> > >>> https://samza.apache.org/learn/documentation/latest/
> > operations/monitoring.html#customreporter
> > >>> > >> > >> >
> > >>> > >> > >>
> > >>> > >> > >> Thanks,
> > >>> > >> > >> Bharath
> > >>> > >> > >>
> > >>> > >> > >> On Wed, Feb 26, 2020 at 3:15 AM Vaibhav Garg <
> > >>> > >> vaibhavgarg90@gmail.com>
> > >>> > >> > >> wrote:
> > >>> > >> > >>
> > >>> > >> > >> > Hi,
> > >>> > >> > >> >
> > >>> > >> > >> > I want to integrate New Relic in my Samza jobs. Can
> anyone
> > >>> help
> > >>> > >> with
> > >>> > >> > >> this,
> > >>> > >> > >> > please?
> > >>> > >> > >> >
> > >>> > >> > >> > Thanks,
> > >>> > >> > >> > Vaibhav Garg
> > >>> > >> > >> > +91-9505020924
> > >>> > >> > >> > vaibhavgarg90@gmail.com
> > >>> > >> > >> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> > >>> > >> > >> >
> > >>> > >> > >>
> > >>> > >> > >
> > >>> > >> >
> > >>> > >>
> > >>> > >
> > >>> >
> > >>>
> > >>
> >
>
>
> --
> Jagadish
>
Re: Samza - New Relic integration
Posted by Jagadish Venkatraman <ja...@gmail.com>.
Hi Vaibhav,
Can you try ./__package/newrelic/newrelic.jar?
If that doesn't work, I'd recommend doing the following:
1. Print the current working directory of your Samza container JVM
2. Use (1) to determine a relative reference to the directory path within
your package
Cheers
Jagadish
On Saturday, February 29, 2020, Vaibhav Garg <va...@gmail.com>
wrote:
> I would be really grateful if someone could help me with this.
>
> Thanks,
> Vaibhav Garg
> +91-9505020924
> vaibhavgarg90@gmail.com
> LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
>
>
> On Sat, Feb 29, 2020 at 9:02 PM Vaibhav Garg <va...@gmail.com>
> wrote:
>
> > Please do let me know if some other clarification is needed.
> >
> > Thanks,
> > Vaibhav Garg
> > +91-9505020924
> > vaibhavgarg90@gmail.com
> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> >
> >
> > On Sat, Feb 29, 2020 at 10:31 AM Vaibhav Garg <va...@gmail.com>
> > wrote:
> >
> >> OK. Here is what I have inside my configuration in order to start
> >> NewRelic at the time of the container start:
> >>
> >> task.opts=-Xmx2048m -XX:+UseConcMarkSweepGC
> >> *-javaagent:/usr/home/vaibhav/newrelic/newrelic.jar*
> >>
> >> Now, I have specified the absolute path of the NewRelic jar. There are 2
> >> challenges that I see with this approach:
> >>
> >> 1. The NewRelic jar and config would have to be present on all the Yarn
> >> slave nodes since that container can start at any of the Yarn nodes.
> >> 2. A single Yarn cluster can be used to run multiple Samza applications,
> >> so I would have to keep configuration for each application at each of
> the
> >> Yarn slave nodes.
> >>
> >> In order to fix this, I am bundling NewRelic inside my tar file:
> >>
> >> // download new relic dependencies
> >> task downloadNewrelic(type: Download) {
> >> mkdir "newrelic"
> >> src "
> >> https://download.newrelic.com/newrelic/java-agent/newrelic-
> agent/current/newrelic-java.zip
> >> "
> >> dest file("newrelic")
> >> }
> >>
> >> task buildTar(dependsOn: [build, downloadNewrelic], type: Tar) {
> >> // unzip new relic dependencies and add to the archive
> >> into(".") {
> >> from zipTree(file("newrelic/newrelic-java.zip"))
> >> }
> >> }
> >>
> >> With this build configuration, I have an additional *newrelic* directory
> >> inside my archive. Here is the directory structure:
> >> my-job-name.tar.gz
> >> bin
> >> config
> >> lib
> >> newrelic
> >> newrelic.jar
> >> newrelic.yml
> >> ...
> >>
> >> What I would like to achieve is to:
> >> - Remove hardcoding the *javaagent* inside *task.opts* in the container
> >> config.
> >> - Specify the path of NewRelic agent in *task.opts* relative to where
> >> the container files are present on the Yarn slave node.
> >>
> >> I hope I was clearer this time. Please let me know if any other
> >> clarification is needed.
> >>
> >> Thanks,
> >> Vaibhav Garg
> >> +91-9505020924
> >> vaibhavgarg90@gmail.com
> >> LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> >>
> >>
> >> On Sat, Feb 29, 2020 at 5:22 AM Yi Pan <ni...@gmail.com> wrote:
> >>
> >>> Hi, Vaibhav,
> >>>
> >>> Not quite sure whether I understand your ask. How do you package and
> >>> distribute NewRelic jar in the YARN cluster? It would be much helpful
> if
> >>> you can share your job configure that worked and the places that you
> >>> would
> >>> like to change.
> >>>
> >>> Best,
> >>>
> >>> -Yi
> >>>
> >>> On Fri, Feb 28, 2020 at 11:55 AM Vaibhav Garg <vaibhavgarg90@gmail.com
> >
> >>> wrote:
> >>>
> >>> > OK. I have been able to use NewRelic by specifying -javaagent option
> in
> >>> > task.opts.
> >>> >
> >>> > However, I had to give the absolute path to the NewRelic jar file in
> >>> order
> >>> > to make it work.
> >>> >
> >>> > How can I give a path that is relative to the container since I would
> >>> like
> >>> > to integrate NewRelic dependencies along with other container files?
> >>> >
> >>> > Thanks,
> >>> > Vaibhav Garg
> >>> > +91-9505020924
> >>> > vaibhavgarg90@gmail.com
> >>> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> >>> >
> >>> >
> >>> > On Sat, Feb 29, 2020 at 12:41 AM Vaibhav Garg <
> vaibhavgarg90@gmail.com
> >>> >
> >>> > wrote:
> >>> >
> >>> > > Yes. That's exactly what I am looking for.
> >>> > >
> >>> > > Let me try it out and I will get back to you if I face any issues.
> >>> > >
> >>> > > Thanks again
> >>> > >
> >>> > > Vaibhav Garg
> >>> > > +91-9505020924
> >>> > > vaibhavgarg90@gmail.com
> >>> > > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> >>> > >
> >>> > >
> >>> > > On Fri, Feb 28, 2020 at 1:23 PM Yi Pan <ni...@gmail.com>
> wrote:
> >>> > >
> >>> > >> Hi, Vaibhav,
> >>> > >>
> >>> > >> Check the description of task.opts in the configuration doc here:
> >>> > >>
> >>> > >>
> >>> >
> >>> http://samza.apache.org/learn/documentation/latest/jobs/
> samza-configurations.html
> >>> > >>
> >>> > >> Is this what you are looking for?
> >>> > >>
> >>> > >> On Thu, Feb 27, 2020 at 9:54 PM Vaibhav Garg <
> >>> vaibhavgarg90@gmail.com>
> >>> > >> wrote:
> >>> > >>
> >>> > >> > Hi,
> >>> > >> >
> >>> > >> > Any help here would be great.
> >>> > >> >
> >>> > >> > Thanks,
> >>> > >> > Vaibhav Garg
> >>> > >> > +91-9505020924
> >>> > >> > vaibhavgarg90@gmail.com
> >>> > >> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> >>> > >> >
> >>> > >> >
> >>> > >> > On Thu, Feb 27, 2020 at 10:20 AM Vaibhav Garg <
> >>> > vaibhavgarg90@gmail.com>
> >>> > >> > wrote:
> >>> > >> >
> >>> > >> > > Dear Bharath,
> >>> > >> > >
> >>> > >> > > Thanks for your reply. I now realize that I put a very
> ambiguous
> >>> > >> question
> >>> > >> > > to the community. Here is another attempt:
> >>> > >> > >
> >>> > >> > > I have set up a Yarn cluster that is configured to run
> >>> containers of
> >>> > >> > > multiple Samza jobs.
> >>> > >> > >
> >>> > >> > > I would like to treat all the containers of the job as a
> single
> >>> > >> > > application in New Relic (as expected).
> >>> > >> > >
> >>> > >> > > Since the containers of the application can be killed on one
> >>> Yarn
> >>> > node
> >>> > >> > and
> >>> > >> > > can start on another Yarn node, I would have to specify New
> >>> Relic
> >>> > >> > > environment settings at the container start.
> >>> > >> > >
> >>> > >> > > Now, I am not sure how to specify java agent (to include New
> >>> Relic
> >>> > >> java
> >>> > >> > > agent) and pass additional arguments such as New Relic
> >>> environment
> >>> > >> every
> >>> > >> > > time the container starts.
> >>> > >> > >
> >>> > >> > > Please let me know if my understanding is wrong or some other
> >>> > >> > > clarification is needed.
> >>> > >> > >
> >>> > >> > > Thanks in advance,
> >>> > >> > > Vaibhav Garg
> >>> > >> > > +91-9505020924
> >>> > >> > > vaibhavgarg90@gmail.com
> >>> > >> > > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> >>> > >> > >
> >>> > >> > >
> >>> > >> > > On Thu, Feb 27, 2020 at 8:25 AM Bharath Kumara Subramanian <
> >>> > >> > > codin.martial@gmail.com> wrote:
> >>> > >> > >
> >>> > >> > >> Hi,
> >>> > >> > >>
> >>> > >> > >> I am not sure I fully understand the ask.
> >>> > >> > >>
> >>> > >> > >> IIUC, Samza doesn't have native integration with New relic.
> >>> > However,
> >>> > >> you
> >>> > >> > >> should still be able to integrate your application with New
> >>> relic
> >>> > on
> >>> > >> > your
> >>> > >> > >> end without native support.
> >>> > >> > >> If you are particularly looking to integrate native Samza
> >>> metrics
> >>> > w/
> >>> > >> New
> >>> > >> > >> relic, you might need to implement your own custom metrics
> >>> > reporter.
> >>> > >> You
> >>> > >> > >> can find more details here
> >>> > >> > >> <
> >>> > >> > >>
> >>> > >> >
> >>> > >>
> >>> >
> >>> https://samza.apache.org/learn/documentation/latest/
> operations/monitoring.html#customreporter
> >>> > >> > >> >
> >>> > >> > >>
> >>> > >> > >> Thanks,
> >>> > >> > >> Bharath
> >>> > >> > >>
> >>> > >> > >> On Wed, Feb 26, 2020 at 3:15 AM Vaibhav Garg <
> >>> > >> vaibhavgarg90@gmail.com>
> >>> > >> > >> wrote:
> >>> > >> > >>
> >>> > >> > >> > Hi,
> >>> > >> > >> >
> >>> > >> > >> > I want to integrate New Relic in my Samza jobs. Can anyone
> >>> help
> >>> > >> with
> >>> > >> > >> this,
> >>> > >> > >> > please?
> >>> > >> > >> >
> >>> > >> > >> > Thanks,
> >>> > >> > >> > Vaibhav Garg
> >>> > >> > >> > +91-9505020924
> >>> > >> > >> > vaibhavgarg90@gmail.com
> >>> > >> > >> > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/>
> >>> > >> > >> >
> >>> > >> > >>
> >>> > >> > >
> >>> > >> >
> >>> > >>
> >>> > >
> >>> >
> >>>
> >>
>
--
Jagadish