You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by "Ganelin, Ilya" <Il...@capitalone.com> on 2015/11/17 23:52:18 UTC
Creating a common metrics server for internal monitoring
Hi, all – presently Apex supports surfacing metrics via the AutoMetrics interface, documented here:
http://docs.datatorrent.com/autometrics/
However, this only provides utility to monitor operators. I would like to provide a more general latency suite - fFor example, I would like to instrument metrics around the BufferServer or the CircularQueue. Not everything in Apex is an operator, and there are other components of the system around which we may want explicit measurements or metrics. Is this something that could exist? I know other projects like Spark have similar efforts, although they are equally underdeveloped :)
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Creating a common metrics server for internal monitoring
Posted by Amol Kekre <am...@datatorrent.com>.
Ilya,
This is a good ask. We should do so. Apex does have system metrics, but
they are very much operator oriented. Adding system metrics makes sense,
including expanding them to include BufferServer, CircularBuffer etc.
https://www.datatorrent.com/docs/3.1.0/guides/DTGatewayAPISpecification.html
Thks
Amol
On Tue, Nov 17, 2015 at 2:52 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Hi, all – presently Apex supports surfacing metrics via the AutoMetrics
> interface, documented here:
> http://docs.datatorrent.com/autometrics/
>
> However, this only provides utility to monitor operators. I would like to
> provide a more general latency suite - fFor example, I would like to
> instrument metrics around the BufferServer or the CircularQueue. Not
> everything in Apex is an operator, and there are other components of the
> system around which we may want explicit measurements or metrics. Is this
> something that could exist? I know other projects like Spark have similar
> efforts, although they are equally underdeveloped :)
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
Re: Creating a common metrics server for internal monitoring
Posted by Munagala Ramanath <ra...@datatorrent.com>.
Adding some concrete data to Pramod's and David's points, entering a
URL like this on the Sandbox:
http://localhost:9090/ws/v2/applications/application_1447787413079_0001
yields output like this:
{"id":"application_1447787413079_0001", ....,
"totalBufferServerReadBytesPSMA":"190","totalBufferServerWriteBytesPSMA":"135",
... }
That's buffer server bytes read and written "Per Second Moving Average".
Ram
On Tue, Nov 17, 2015 at 2:58 PM, Pramod Immaneni <pr...@datatorrent.com> wrote:
> Some of them are available in system stats like port queue size and are
> exposed via REST and to a stats listener that you can register via the API.
> Some of the others you mentioned can be added.
>
> On Tue, Nov 17, 2015 at 2:52 PM, Ganelin, Ilya <Il...@capitalone.com>
> wrote:
>
>> Hi, all – presently Apex supports surfacing metrics via the AutoMetrics
>> interface, documented here:
>> http://docs.datatorrent.com/autometrics/
>>
>> However, this only provides utility to monitor operators. I would like to
>> provide a more general latency suite - fFor example, I would like to
>> instrument metrics around the BufferServer or the CircularQueue. Not
>> everything in Apex is an operator, and there are other components of the
>> system around which we may want explicit measurements or metrics. Is this
>> something that could exist? I know other projects like Spark have similar
>> efforts, although they are equally underdeveloped :)
>> ________________________________________________________
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The information
>> transmitted herewith is intended only for use by the individual or entity
>> to which it is addressed. If the reader of this message is not the intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
Re: Creating a common metrics server for internal monitoring
Posted by David Yan <da...@datatorrent.com>.
Hmm... if we somehow mark a tuple as a "test" tuple or a "pilot" tuple, I
think we can do the same thing on it with the END_WINDOW tuple to calculate
the latency.
However, if there are many such tuples, it will for sure incur significant
cost and affect the latency itself, kinda like the Heisenberg's principle
-- you can't accurately measure an object because doing so will alter the
object itself.
David
On Tue, Nov 17, 2015 at 4:12 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> I totally agree - I think any such functionality should be a configurable
> setting, and should not necessarily apply to every single tuple but
> perhaps be periodically computed for “test” tuples flowing through the
> system.
>
>
> On 11/17/15, 4:10 PM, "Pramod Immaneni" <pr...@datatorrent.com> wrote:
>
> >I am wondering if accounting latency at a tuple level will add too much
> >overhead. It might be better to do it on a conditional basis.
> >
> >On Tue, Nov 17, 2015 at 3:48 PM, Ganelin, Ilya
> ><Il...@capitalone.com>
> >wrote:
> >
> >> Specifically, I¹m interested in being able to track latencies for
> >> individual tuples. For example, as they move through the BufferServer or
> >> the CircularBuffer.
> >>
> >> On 11/17/15, 3:12 PM, "David Yan" <da...@datatorrent.com> wrote:
> >>
> >> >Hi Ilya,
> >> >
> >> >The AutoMetric interface is mostly designed for application-specific
> >> >metrics within an operator -- things like how many frauds are
> >>detected, or
> >> >the dollar amount processed.
> >> >
> >> >We do have BufferServer bytes and queue tuple count (a.k.a. queue
> >>size) as
> >> >system metrics. If you have specific metrics that pertain to
> >>BufferServer
> >> >and CircularBuffer that you're interested in, please state them. We
> >> >should
> >> >probably add them to system metrics since it's probably useful in
> >>general.
> >> >
> >> >David
> >> >
> >> >On Tue, Nov 17, 2015 at 2:58 PM, Pramod Immaneni
> >><pr...@datatorrent.com>
> >> >wrote:
> >> >
> >> >> Some of them are available in system stats like port queue size and
> >>are
> >> >> exposed via REST and to a stats listener that you can register via
> >>the
> >> >>API.
> >> >> Some of the others you mentioned can be added.
> >> >>
> >> >> On Tue, Nov 17, 2015 at 2:52 PM, Ganelin, Ilya <
> >> >> Ilya.Ganelin@capitalone.com>
> >> >> wrote:
> >> >>
> >> >> > Hi, all presently Apex supports surfacing metrics via the
> >> >>AutoMetrics
> >> >> > interface, documented here:
> >> >> > http://docs.datatorrent.com/autometrics/
> >> >> >
> >> >> > However, this only provides utility to monitor operators. I would
> >> >>like to
> >> >> > provide a more general latency suite - fFor example, I would like
> >>to
> >> >> > instrument metrics around the BufferServer or the CircularQueue.
> >>Not
> >> >> > everything in Apex is an operator, and there are other components
> >>of
> >> >>the
> >> >> > system around which we may want explicit measurements or metrics.
> >>Is
> >> >>this
> >> >> > something that could exist? I know other projects like Spark have
> >> >>similar
> >> >> > efforts, although they are equally underdeveloped :)
> >> >> > ________________________________________________________
> >> >> >
> >> >> > The information contained in this e-mail is confidential and/or
> >> >> > proprietary to Capital One and/or its affiliates and may only be
> >>used
> >> >> > solely in performance of work or services for Capital One. The
> >> >> information
> >> >> > transmitted herewith is intended only for use by the individual or
> >> >>entity
> >> >> > to which it is addressed. If the reader of this message is not the
> >> >> intended
> >> >> > recipient, you are hereby notified that any review, retransmission,
> >> >> > dissemination, distribution, copying or other use of, or taking of
> >>any
> >> >> > action in reliance upon this information is strictly prohibited. If
> >> >>you
> >> >> > have received this communication in error, please contact the
> >>sender
> >> >>and
> >> >> > delete the material from your computer.
> >> >> >
> >> >>
> >>
> >> ________________________________________________________
> >>
> >> The information contained in this e-mail is confidential and/or
> >> proprietary to Capital One and/or its affiliates and may only be used
> >> solely in performance of work or services for Capital One. The
> >>information
> >> transmitted herewith is intended only for use by the individual or
> >>entity
> >> to which it is addressed. If the reader of this message is not the
> >>intended
> >> recipient, you are hereby notified that any review, retransmission,
> >> dissemination, distribution, copying or other use of, or taking of any
> >> action in reliance upon this information is strictly prohibited. If you
> >> have received this communication in error, please contact the sender and
> >> delete the material from your computer.
> >>
> >>
>
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
Re: Creating a common metrics server for internal monitoring
Posted by "Ganelin, Ilya" <Il...@capitalone.com>.
I totally agree - I think any such functionality should be a configurable
setting, and should not necessarily apply to every single tuple but
perhaps be periodically computed for “test” tuples flowing through the
system.
On 11/17/15, 4:10 PM, "Pramod Immaneni" <pr...@datatorrent.com> wrote:
>I am wondering if accounting latency at a tuple level will add too much
>overhead. It might be better to do it on a conditional basis.
>
>On Tue, Nov 17, 2015 at 3:48 PM, Ganelin, Ilya
><Il...@capitalone.com>
>wrote:
>
>> Specifically, I¹m interested in being able to track latencies for
>> individual tuples. For example, as they move through the BufferServer or
>> the CircularBuffer.
>>
>> On 11/17/15, 3:12 PM, "David Yan" <da...@datatorrent.com> wrote:
>>
>> >Hi Ilya,
>> >
>> >The AutoMetric interface is mostly designed for application-specific
>> >metrics within an operator -- things like how many frauds are
>>detected, or
>> >the dollar amount processed.
>> >
>> >We do have BufferServer bytes and queue tuple count (a.k.a. queue
>>size) as
>> >system metrics. If you have specific metrics that pertain to
>>BufferServer
>> >and CircularBuffer that you're interested in, please state them. We
>> >should
>> >probably add them to system metrics since it's probably useful in
>>general.
>> >
>> >David
>> >
>> >On Tue, Nov 17, 2015 at 2:58 PM, Pramod Immaneni
>><pr...@datatorrent.com>
>> >wrote:
>> >
>> >> Some of them are available in system stats like port queue size and
>>are
>> >> exposed via REST and to a stats listener that you can register via
>>the
>> >>API.
>> >> Some of the others you mentioned can be added.
>> >>
>> >> On Tue, Nov 17, 2015 at 2:52 PM, Ganelin, Ilya <
>> >> Ilya.Ganelin@capitalone.com>
>> >> wrote:
>> >>
>> >> > Hi, all presently Apex supports surfacing metrics via the
>> >>AutoMetrics
>> >> > interface, documented here:
>> >> > http://docs.datatorrent.com/autometrics/
>> >> >
>> >> > However, this only provides utility to monitor operators. I would
>> >>like to
>> >> > provide a more general latency suite - fFor example, I would like
>>to
>> >> > instrument metrics around the BufferServer or the CircularQueue.
>>Not
>> >> > everything in Apex is an operator, and there are other components
>>of
>> >>the
>> >> > system around which we may want explicit measurements or metrics.
>>Is
>> >>this
>> >> > something that could exist? I know other projects like Spark have
>> >>similar
>> >> > efforts, although they are equally underdeveloped :)
>> >> > ________________________________________________________
>> >> >
>> >> > The information contained in this e-mail is confidential and/or
>> >> > proprietary to Capital One and/or its affiliates and may only be
>>used
>> >> > solely in performance of work or services for Capital One. The
>> >> information
>> >> > transmitted herewith is intended only for use by the individual or
>> >>entity
>> >> > to which it is addressed. If the reader of this message is not the
>> >> intended
>> >> > recipient, you are hereby notified that any review, retransmission,
>> >> > dissemination, distribution, copying or other use of, or taking of
>>any
>> >> > action in reliance upon this information is strictly prohibited. If
>> >>you
>> >> > have received this communication in error, please contact the
>>sender
>> >>and
>> >> > delete the material from your computer.
>> >> >
>> >>
>>
>> ________________________________________________________
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The
>>information
>> transmitted herewith is intended only for use by the individual or
>>entity
>> to which it is addressed. If the reader of this message is not the
>>intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
>>
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Creating a common metrics server for internal monitoring
Posted by Pramod Immaneni <pr...@datatorrent.com>.
I am wondering if accounting latency at a tuple level will add too much
overhead. It might be better to do it on a conditional basis.
On Tue, Nov 17, 2015 at 3:48 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Specifically, I¹m interested in being able to track latencies for
> individual tuples. For example, as they move through the BufferServer or
> the CircularBuffer.
>
> On 11/17/15, 3:12 PM, "David Yan" <da...@datatorrent.com> wrote:
>
> >Hi Ilya,
> >
> >The AutoMetric interface is mostly designed for application-specific
> >metrics within an operator -- things like how many frauds are detected, or
> >the dollar amount processed.
> >
> >We do have BufferServer bytes and queue tuple count (a.k.a. queue size) as
> >system metrics. If you have specific metrics that pertain to BufferServer
> >and CircularBuffer that you're interested in, please state them. We
> >should
> >probably add them to system metrics since it's probably useful in general.
> >
> >David
> >
> >On Tue, Nov 17, 2015 at 2:58 PM, Pramod Immaneni <pr...@datatorrent.com>
> >wrote:
> >
> >> Some of them are available in system stats like port queue size and are
> >> exposed via REST and to a stats listener that you can register via the
> >>API.
> >> Some of the others you mentioned can be added.
> >>
> >> On Tue, Nov 17, 2015 at 2:52 PM, Ganelin, Ilya <
> >> Ilya.Ganelin@capitalone.com>
> >> wrote:
> >>
> >> > Hi, all presently Apex supports surfacing metrics via the
> >>AutoMetrics
> >> > interface, documented here:
> >> > http://docs.datatorrent.com/autometrics/
> >> >
> >> > However, this only provides utility to monitor operators. I would
> >>like to
> >> > provide a more general latency suite - fFor example, I would like to
> >> > instrument metrics around the BufferServer or the CircularQueue. Not
> >> > everything in Apex is an operator, and there are other components of
> >>the
> >> > system around which we may want explicit measurements or metrics. Is
> >>this
> >> > something that could exist? I know other projects like Spark have
> >>similar
> >> > efforts, although they are equally underdeveloped :)
> >> > ________________________________________________________
> >> >
> >> > The information contained in this e-mail is confidential and/or
> >> > proprietary to Capital One and/or its affiliates and may only be used
> >> > solely in performance of work or services for Capital One. The
> >> information
> >> > transmitted herewith is intended only for use by the individual or
> >>entity
> >> > to which it is addressed. If the reader of this message is not the
> >> intended
> >> > recipient, you are hereby notified that any review, retransmission,
> >> > dissemination, distribution, copying or other use of, or taking of any
> >> > action in reliance upon this information is strictly prohibited. If
> >>you
> >> > have received this communication in error, please contact the sender
> >>and
> >> > delete the material from your computer.
> >> >
> >>
>
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
>
Re: Creating a common metrics server for internal monitoring
Posted by "Ganelin, Ilya" <Il...@capitalone.com>.
David - that’s a correct interpretation, although we’re interested in
having a measurement for every tuple, rather than just our aggregate. For
our use case we actually require the ability to track any individual tuple
end-end through the entire system. We are aware that it’s possible to
measure latency on a window basis but this does not provide sufficient
granularity. The main purpose of this is to be able to conclusively
identify bottlenecks and limitations within the Apex architecture.
On 11/17/15, 4:07 PM, "David Yan" <da...@datatorrent.com> wrote:
>Hi Ilya,
>
>Just wanted to make sure I understand correctly. When you said latencies
>for individual tuples, do you mean how many milliseconds on average a
>tuple
>stays in the queue in the upstream operator before
>DefaultInputPort.process() is called in the current Operator?
>
>We currently have latency per operator based on the endWindow tuple
>dequeueing time. Although this is not the same, it gives you an idea how
>much latency an operator imposes on the stream on a per-window basis.
>Example:
>
>$ curl
>http://node0.morado.com:9099/ws/v2/applications/application_1447661686850_
>17918/physicalPlan/operators/11
>| python -mjson.tool
>{
> ...
> "latencyMA": "287",
> ...
>}
>
>David
>
>
>On Tue, Nov 17, 2015 at 3:48 PM, Ganelin, Ilya
><Il...@capitalone.com>
>wrote:
>
>> Specifically, I¹m interested in being able to track latencies for
>> individual tuples. For example, as they move through the BufferServer or
>> the CircularBuffer.
>>
>> On 11/17/15, 3:12 PM, "David Yan" <da...@datatorrent.com> wrote:
>>
>> >Hi Ilya,
>> >
>> >The AutoMetric interface is mostly designed for application-specific
>> >metrics within an operator -- things like how many frauds are
>>detected, or
>> >the dollar amount processed.
>> >
>> >We do have BufferServer bytes and queue tuple count (a.k.a. queue
>>size) as
>> >system metrics. If you have specific metrics that pertain to
>>BufferServer
>> >and CircularBuffer that you're interested in, please state them. We
>> >should
>> >probably add them to system metrics since it's probably useful in
>>general.
>> >
>> >David
>> >
>> >On Tue, Nov 17, 2015 at 2:58 PM, Pramod Immaneni
>><pr...@datatorrent.com>
>> >wrote:
>> >
>> >> Some of them are available in system stats like port queue size and
>>are
>> >> exposed via REST and to a stats listener that you can register via
>>the
>> >>API.
>> >> Some of the others you mentioned can be added.
>> >>
>> >> On Tue, Nov 17, 2015 at 2:52 PM, Ganelin, Ilya <
>> >> Ilya.Ganelin@capitalone.com>
>> >> wrote:
>> >>
>> >> > Hi, all presently Apex supports surfacing metrics via the
>> >>AutoMetrics
>> >> > interface, documented here:
>> >> > http://docs.datatorrent.com/autometrics/
>> >> >
>> >> > However, this only provides utility to monitor operators. I would
>> >>like to
>> >> > provide a more general latency suite - fFor example, I would like
>>to
>> >> > instrument metrics around the BufferServer or the CircularQueue.
>>Not
>> >> > everything in Apex is an operator, and there are other components
>>of
>> >>the
>> >> > system around which we may want explicit measurements or metrics.
>>Is
>> >>this
>> >> > something that could exist? I know other projects like Spark have
>> >>similar
>> >> > efforts, although they are equally underdeveloped :)
>> >> > ________________________________________________________
>> >> >
>> >> > The information contained in this e-mail is confidential and/or
>> >> > proprietary to Capital One and/or its affiliates and may only be
>>used
>> >> > solely in performance of work or services for Capital One. The
>> >> information
>> >> > transmitted herewith is intended only for use by the individual or
>> >>entity
>> >> > to which it is addressed. If the reader of this message is not the
>> >> intended
>> >> > recipient, you are hereby notified that any review, retransmission,
>> >> > dissemination, distribution, copying or other use of, or taking of
>>any
>> >> > action in reliance upon this information is strictly prohibited. If
>> >>you
>> >> > have received this communication in error, please contact the
>>sender
>> >>and
>> >> > delete the material from your computer.
>> >> >
>> >>
>>
>> ________________________________________________________
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The
>>information
>> transmitted herewith is intended only for use by the individual or
>>entity
>> to which it is addressed. If the reader of this message is not the
>>intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
>>
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Creating a common metrics server for internal monitoring
Posted by David Yan <da...@datatorrent.com>.
Hi Ilya,
Just wanted to make sure I understand correctly. When you said latencies
for individual tuples, do you mean how many milliseconds on average a tuple
stays in the queue in the upstream operator before
DefaultInputPort.process() is called in the current Operator?
We currently have latency per operator based on the endWindow tuple
dequeueing time. Although this is not the same, it gives you an idea how
much latency an operator imposes on the stream on a per-window basis.
Example:
$ curl
http://node0.morado.com:9099/ws/v2/applications/application_1447661686850_17918/physicalPlan/operators/11
| python -mjson.tool
{
...
"latencyMA": "287",
...
}
David
On Tue, Nov 17, 2015 at 3:48 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Specifically, I¹m interested in being able to track latencies for
> individual tuples. For example, as they move through the BufferServer or
> the CircularBuffer.
>
> On 11/17/15, 3:12 PM, "David Yan" <da...@datatorrent.com> wrote:
>
> >Hi Ilya,
> >
> >The AutoMetric interface is mostly designed for application-specific
> >metrics within an operator -- things like how many frauds are detected, or
> >the dollar amount processed.
> >
> >We do have BufferServer bytes and queue tuple count (a.k.a. queue size) as
> >system metrics. If you have specific metrics that pertain to BufferServer
> >and CircularBuffer that you're interested in, please state them. We
> >should
> >probably add them to system metrics since it's probably useful in general.
> >
> >David
> >
> >On Tue, Nov 17, 2015 at 2:58 PM, Pramod Immaneni <pr...@datatorrent.com>
> >wrote:
> >
> >> Some of them are available in system stats like port queue size and are
> >> exposed via REST and to a stats listener that you can register via the
> >>API.
> >> Some of the others you mentioned can be added.
> >>
> >> On Tue, Nov 17, 2015 at 2:52 PM, Ganelin, Ilya <
> >> Ilya.Ganelin@capitalone.com>
> >> wrote:
> >>
> >> > Hi, all presently Apex supports surfacing metrics via the
> >>AutoMetrics
> >> > interface, documented here:
> >> > http://docs.datatorrent.com/autometrics/
> >> >
> >> > However, this only provides utility to monitor operators. I would
> >>like to
> >> > provide a more general latency suite - fFor example, I would like to
> >> > instrument metrics around the BufferServer or the CircularQueue. Not
> >> > everything in Apex is an operator, and there are other components of
> >>the
> >> > system around which we may want explicit measurements or metrics. Is
> >>this
> >> > something that could exist? I know other projects like Spark have
> >>similar
> >> > efforts, although they are equally underdeveloped :)
> >> > ________________________________________________________
> >> >
> >> > The information contained in this e-mail is confidential and/or
> >> > proprietary to Capital One and/or its affiliates and may only be used
> >> > solely in performance of work or services for Capital One. The
> >> information
> >> > transmitted herewith is intended only for use by the individual or
> >>entity
> >> > to which it is addressed. If the reader of this message is not the
> >> intended
> >> > recipient, you are hereby notified that any review, retransmission,
> >> > dissemination, distribution, copying or other use of, or taking of any
> >> > action in reliance upon this information is strictly prohibited. If
> >>you
> >> > have received this communication in error, please contact the sender
> >>and
> >> > delete the material from your computer.
> >> >
> >>
>
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
>
Re: Creating a common metrics server for internal monitoring
Posted by "Ganelin, Ilya" <Il...@capitalone.com>.
Specifically, I¹m interested in being able to track latencies for
individual tuples. For example, as they move through the BufferServer or
the CircularBuffer.
On 11/17/15, 3:12 PM, "David Yan" <da...@datatorrent.com> wrote:
>Hi Ilya,
>
>The AutoMetric interface is mostly designed for application-specific
>metrics within an operator -- things like how many frauds are detected, or
>the dollar amount processed.
>
>We do have BufferServer bytes and queue tuple count (a.k.a. queue size) as
>system metrics. If you have specific metrics that pertain to BufferServer
>and CircularBuffer that you're interested in, please state them. We
>should
>probably add them to system metrics since it's probably useful in general.
>
>David
>
>On Tue, Nov 17, 2015 at 2:58 PM, Pramod Immaneni <pr...@datatorrent.com>
>wrote:
>
>> Some of them are available in system stats like port queue size and are
>> exposed via REST and to a stats listener that you can register via the
>>API.
>> Some of the others you mentioned can be added.
>>
>> On Tue, Nov 17, 2015 at 2:52 PM, Ganelin, Ilya <
>> Ilya.Ganelin@capitalone.com>
>> wrote:
>>
>> > Hi, all presently Apex supports surfacing metrics via the
>>AutoMetrics
>> > interface, documented here:
>> > http://docs.datatorrent.com/autometrics/
>> >
>> > However, this only provides utility to monitor operators. I would
>>like to
>> > provide a more general latency suite - fFor example, I would like to
>> > instrument metrics around the BufferServer or the CircularQueue. Not
>> > everything in Apex is an operator, and there are other components of
>>the
>> > system around which we may want explicit measurements or metrics. Is
>>this
>> > something that could exist? I know other projects like Spark have
>>similar
>> > efforts, although they are equally underdeveloped :)
>> > ________________________________________________________
>> >
>> > The information contained in this e-mail is confidential and/or
>> > proprietary to Capital One and/or its affiliates and may only be used
>> > solely in performance of work or services for Capital One. The
>> information
>> > transmitted herewith is intended only for use by the individual or
>>entity
>> > to which it is addressed. If the reader of this message is not the
>> intended
>> > recipient, you are hereby notified that any review, retransmission,
>> > dissemination, distribution, copying or other use of, or taking of any
>> > action in reliance upon this information is strictly prohibited. If
>>you
>> > have received this communication in error, please contact the sender
>>and
>> > delete the material from your computer.
>> >
>>
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Creating a common metrics server for internal monitoring
Posted by David Yan <da...@datatorrent.com>.
Hi Ilya,
The AutoMetric interface is mostly designed for application-specific
metrics within an operator -- things like how many frauds are detected, or
the dollar amount processed.
We do have BufferServer bytes and queue tuple count (a.k.a. queue size) as
system metrics. If you have specific metrics that pertain to BufferServer
and CircularBuffer that you're interested in, please state them. We should
probably add them to system metrics since it's probably useful in general.
David
On Tue, Nov 17, 2015 at 2:58 PM, Pramod Immaneni <pr...@datatorrent.com>
wrote:
> Some of them are available in system stats like port queue size and are
> exposed via REST and to a stats listener that you can register via the API.
> Some of the others you mentioned can be added.
>
> On Tue, Nov 17, 2015 at 2:52 PM, Ganelin, Ilya <
> Ilya.Ganelin@capitalone.com>
> wrote:
>
> > Hi, all – presently Apex supports surfacing metrics via the AutoMetrics
> > interface, documented here:
> > http://docs.datatorrent.com/autometrics/
> >
> > However, this only provides utility to monitor operators. I would like to
> > provide a more general latency suite - fFor example, I would like to
> > instrument metrics around the BufferServer or the CircularQueue. Not
> > everything in Apex is an operator, and there are other components of the
> > system around which we may want explicit measurements or metrics. Is this
> > something that could exist? I know other projects like Spark have similar
> > efforts, although they are equally underdeveloped :)
> > ________________________________________________________
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >
>
Re: Creating a common metrics server for internal monitoring
Posted by Pramod Immaneni <pr...@datatorrent.com>.
Some of them are available in system stats like port queue size and are
exposed via REST and to a stats listener that you can register via the API.
Some of the others you mentioned can be added.
On Tue, Nov 17, 2015 at 2:52 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Hi, all – presently Apex supports surfacing metrics via the AutoMetrics
> interface, documented here:
> http://docs.datatorrent.com/autometrics/
>
> However, this only provides utility to monitor operators. I would like to
> provide a more general latency suite - fFor example, I would like to
> instrument metrics around the BufferServer or the CircularQueue. Not
> everything in Apex is an operator, and there are other components of the
> system around which we may want explicit measurements or metrics. Is this
> something that could exist? I know other projects like Spark have similar
> efforts, although they are equally underdeveloped :)
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>