You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@deltacloud.apache.org by "Koper, Dies" <di...@fast.au.fujitsu.com> on 2012/09/02 05:53:47 UTC

RE: normalised metric names?

Looking at the EC2 Cloudwatch sample Michal once shared with me, I'm
still not sure how to interpret their data and how it should be modelled
in DC.
Let me copy and paste part of the XML he sent me:

<?xml version="1.0" encoding="UTF-8"?>
<metric href="http://localhost:3002/api/metrics/i-5cac1c24"
id="i-5cac1c24">
  <entity>i-5cac1c24</entity>
  <properties>
    <NetworkIn>
    <sample>
      <property name="timestamp" value="2012-07-09T12:38:00Z"/>
      <property name="unit" value="Bytes"/>
      <property name="minimum" value="42.0"/>
      <property name="maximum" value="42.0"/>
      <property name="average" value="42.0"/>
      <property name="samples" value="1.0"/>
    </sample>
    <sample>
      <property name="timestamp" value="2012-07-09T12:34:00Z"/>
      <property name="unit" value="Bytes"/>
      <property name="minimum" value="84.0"/>
      <property name="maximum" value="84.0"/>
      <property name="average" value="84.0"/>
      <property name="samples" value="1.0"/>
    </sample>
    <sample>
...

What I observe is:
1. The 'samples' value is always 1. I suppose that's why a sample's min,
max and average are the same. In what case would samples not be 1?
2. Even though the 'samples' value is 1, clearly it returns multiple
samples (=measurements?) over a period (every 6 min. judging from the
timestamp).

The only useful data I get is the 42 and 84 and their timestamps. FGCP
returns that data too, it just doesn't call them average, min and max.
Do we need to include it three times in the DC API?

        <sample timestamp="..." count="..." unit="...">
          <entry name="average" value="42.0"/>
<!--
          <entry name="minimum" value="42.0"/>
          <entry name="maximum" value="42.0"/>
-->
        </sample>

Regards,
Dies Koper


> -----Original Message-----
> From: David Lutterkort [mailto:lutter@redhat.com]
> Sent: Tuesday, 28 August 2012 10:32 AM
> To: dev@deltacloud.apache.org
> Subject: RE: normalised metric names?
> 
> On Fri, 2012-08-24 at 16:46 +1000, Koper, Dies wrote:
> > Hi David,
> >
> > > normalized name should not contain a unit ("nicInputByte"), and
I'd
> > > prefer if it was underscored, but whether we call it network_in or
> > > nic_input doesn't make a difference to me.
> >
> > I've made a new list (below).
> > I'm suggesting nic instead of network for consistency with cpu (vs.
> > compute).
> > I wonder if that will turn out to be the better choice if providers
> > start returning samples per nic (when a server has several), or not
:)
> >
> > EC2 "provider_probe"	FGCP "provider_probe"
Normalised
> > "probe"
> > CPUUtilization		cpuUtilization
cpu_utilization
> >
> > NetworkIn		nicInputByte			nic_in
> > NetworkOut		nicOutputByte			nic_out
> > -			nicInputPacket			-
> > -			nicOutputPacket		-
> >
> > DiskReadOps		diskReadRequestCount		disk_read_ops
> > DiskWriteOps		diskWriteRequestCount
disk_write_ops
> > DiskReadBytes		-				-
> > DiskWriteBytes		-				-
> > -			diskReadSector			-
> > -			diskWriteSector			-
> 
> Those names work for me.
> 
> > > seems clearer; this would indicate that we took a sample at
> > @timestamp,
> > > consisting of @count measurements, and the values are reported in
> > @unit.
> > > The body of <sample/> only contains the statistics for that one
> > sample.
> > >
> > > What isn't clear to me is how we can talk about taking a sample at
a
> > > certain time when that sample consists of multiple measurements -
> > > shouldn't that sample then have an underlying time period (from
time X
> > > to time Y, we looked N times, and here are aggregate stats of what
we
> > > saw ?)
> >
> > FGCP doesn't return "average", min and max for metrics. It just
returns
> > a number for each.
> > I assumed each sample returned subtotals since the previous
measurement.
> > Except for cpu utilization, not sure how that's determined.
> 
> What EC2 does, is that it takes a number N of samples, and then tells
> you N and aggregate statistics for those N. If FGCP returns
measurements
> directly, we can just indicate that by saying we took 1 measurement
and
> return the value as 'average' (or 'max' or 'min' ;)
> 
> David
> 
> 



RE: normalised metric names?

Posted by David Lutterkort <lu...@redhat.com>.
On Sun, 2012-09-02 at 13:53 +1000, Koper, Dies wrote:
> Looking at the EC2 Cloudwatch sample Michal once shared with me, I'm
> still not sure how to interpret their data and how it should be modelled
> in DC.
> Let me copy and paste part of the XML he sent me:
> 
> <?xml version="1.0" encoding="UTF-8"?>
> <metric href="http://localhost:3002/api/metrics/i-5cac1c24"
> id="i-5cac1c24">
>   <entity>i-5cac1c24</entity>
>   <properties>
>     <NetworkIn>
>     <sample>
>       <property name="timestamp" value="2012-07-09T12:38:00Z"/>
>       <property name="unit" value="Bytes"/>
>       <property name="minimum" value="42.0"/>
>       <property name="maximum" value="42.0"/>
>       <property name="average" value="42.0"/>
>       <property name="samples" value="1.0"/>
>     </sample>
>     <sample>
>       <property name="timestamp" value="2012-07-09T12:34:00Z"/>
>       <property name="unit" value="Bytes"/>
>       <property name="minimum" value="84.0"/>
>       <property name="maximum" value="84.0"/>
>       <property name="average" value="84.0"/>
>       <property name="samples" value="1.0"/>
>     </sample>
>     <sample>
> ...
> 
> What I observe is:
> 1. The 'samples' value is always 1. I suppose that's why a sample's min,
> max and average are the same. In what case would samples not be 1?
> 2. Even though the 'samples' value is 1, clearly it returns multiple
> samples (=measurements?) over a period (every 6 min. judging from the
> timestamp).
> 
> The only useful data I get is the 42 and 84 and their timestamps. FGCP
> returns that data too, it just doesn't call them average, min and max.
> Do we need to include it three times in the DC API?

The important thing to me in the XML format is to prepare the format for
when the backend does only produce aggregates for multiple measurements.
I am not sure under what circumstances EC2 produces metrics for more
than one sample, but I think the general idea is sound: allow the
backend to do some aggregation of samples, and only report certain
staistics for them.

That in the case of nsamples = 1 average, min, and max are redundant is
somewhat unfortunate, and I would be perfectly fine to say that we only
expect backends to report an average, with min and max being optional.

IOW, for FGCP, I'd be fine if the XML looked something like

  <sample timestamp="..." count="1" unit="...">
    <entry name="average" value="42.0"/>
  </sample>

That gives us the flexibility to report more staistics for count > 1.

David