You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@deltacloud.apache.org by samridh srinath <sa...@gmail.com> on 2012/08/06 15:26:50 UTC
Proposal: Adding billing API support to Deltacloud
Hello All,
Let me start by introducing myself. I am Samridh, a student developer part
of Google Summer of Code (GSoC). I have been contributing to the Aeolus
project for the better part of the summer, adding API support for hardware
profiles to the Conductor API. My mentor is Mo Morsi who has been assisting
me.
For what remains of the coding period of GSoC and also after that, Mo and I
thought that it would be a good idea for me to look into adding billing
support to the Deltacloud API. I have also been in conversation with
mfojitk and marios in this regard.
I will be outlining my research and proposal for this in the remaining part
of this mail.
Currently, Amazon ec2 and GoGrid support programmatic retrieval of billing
data though in different formats.
[1] outlines how to retrieve billing data from GoGrid. It returns data in
xml/json/csv via the REST API.
[2] outlines how to activate programmatic billing access for an ec2
account. Essentially, it involves activating programmatic billing access
and then specifying an S3 bucket (either new or an existing bucket) where a
CSV file containing the usage and billing data will be dumped periodically
(in the form of an S3 object). The key for retrieving this S3 object is in
the format "account_number-aws-billing-csv-YYYY-MM.csv".
I would like to propose adding billing API support for these two providers
to start off with and then extending that support as more providers allow
for convenient retrieval of billing data.
This would essentially require the addition of a new collection, namely the
"Billing" collection which has a model that has attributes most common
among all the providers.
The ec2 and gogrid drivers would then be extended to retrieve the billing
data from the respective cloud back-ends.
The CSV data from the ec2 provider can be converted to xml/json with the
help of a rubygem like fastercsv or similar. I am attaching a sample ec2
billing CSV file for your reference (Thanks to Mo for that).
Any suggestions with respect to the billing model and its attributes are
welcome and will definitely help.
Also, any other suggestions regarding its implementation are also welcome.
Looking forward to your feedback and suggestions,
Samridh
Student Developer,
GSoC
[1] https://wiki.gogrid.com/wiki/index.php/API:myaccount.billing.get
[2]
http://docs.amazonwebservices.com/awsaccountbilling/latest/about/programaccess.html
Re: Proposal: Adding billing API support to Deltacloud
Posted by Mark McLoughlin <ma...@redhat.com>.
Hi,
On Mon, 2012-08-06 at 18:56 +0530, samridh srinath wrote:
> Hello All,
>
> Let me start by introducing myself. I am Samridh, a student developer part
> of Google Summer of Code (GSoC). I have been contributing to the Aeolus
> project for the better part of the summer, adding API support for hardware
> profiles to the Conductor API. My mentor is Mo Morsi who has been assisting
> me.
> For what remains of the coding period of GSoC and also after that, Mo and I
> thought that it would be a good idea for me to look into adding billing
> support to the Deltacloud API. I have also been in conversation with
> mfojitk and marios in this regard.
Excellent!
I spent a little while thinking about this some time back and wrote down
some notes quickly. I've added them below FWIW.
The place I'd like to eventually see us get to is that clouds would
provide historical metering data and the cost basis applied to those
meters. So, rather than just "you've spent $200 on instances this week"
you get enough information to calculate the bill yourself.
Now, it looks like AWS is a long way from that but I could see OpenStack
get there soon enough, so it'd be nice if we could design the API to
allow for it in the future.
I guess the summary is there are at least three things worth modelling -
meters, rates and billing.
Cheers,
Mark.
GET /api HTTP/1.1
Accept: application/xml
HTTP/1.1 200 OK
Content-Type: application/xml
<api>
...
<link rel="meters" href="/api/meters?start={start}&end={end}"/>
<link rel="costmodels" href="/api/costmodels"/>
<link rel="currencies" href="/api/currrencies"/>
...
</api>
- meters: current and historical information on resource usage
GET /api/meters?start=2010-10-01&end=2010-10-31 HTTP/1.1
Accept: application/xml
HTTP/1.1 200 OK
Content-Type: application/xml
<meters start="2010-10-01T00:00:00" end="2010-10-01T00:00:00">
<meter id="instance.inst1.uptime" href="/api/meters/instance.inst1.uptime">
<name>Instance1 Running Time</name>
<description>Number of hours Instance1 is running</description>
<value units="hours">42</value>
<instance id="inst1" href="/api/instances/inst1"/>
<link rel="rates" href="/api/meters/instance.inst1.uptime/rates"/>
<link rel="samples" href="/api/meters/instance.inst1.uptime/samples"/>
</meter>
<meter id="instance.inst2.uptime" href="/api/meters/instance.inst2.uptime">
<name>Instance2 Running Time</name>
<description>Number of hours Instance2 is running</description>
<value units="hours">156</value>
<instance id="inst1" href="/api/instances/inst2"/>
<link rel="rates" href="/api/meters/instance.inst2.uptime/rates"/>
<link rel="samples" href="/api/meters/instance.inst2.uptime/samples"/>
</meter>
<meter id="network.rx" href="/api/meters/network.rx">
<name>Network Data In</name>
<description>Number of Gb network data received</description>
<value units="Gb">15.3</value>
<link rel="rates" href="/api/meters/network.rx/rates"/>
<link rel="samples" href="/api/meters/network.rx/samples"/>
</meter>
<meter id="network.tx" href="/api/meters/network.tx">
<name>Network Data Out</name>
<description>Number of Gb network data transmitted</description>
<value units="Gb">38.2</value>
<link rel="rates" href="/api/meters/network.tx/rates"/>
<link rel="samples" href="/api/meters/network.tx/samples"/>
</meter>
<meter id="ebs.provisioned" href="/api/meters/ebs.provisioned">
<name>EBS Storage Provisioned</name>
<description>Number of Gb allocated in EBS storage</description>
<value units="Gb">40.0</value>
<link rel="rates" href="/api/meters/ebs.provisioned/rates"/>
<link rel="samples" href="/api/meters/ebs.provisioned/samples"/>
</meter>
<meter id="ebs.ios" href="/api/meters/ebs.ios">
<name>EBS Storage I/Os</name>
<description>Number of EBS I/O operations</description>
<value>4000000000</value>
<link rel="rates" href="/api/meters/ebs.ios/rates"/>
<link rel="samples" href="/api/meters/ebs.ios/samples"/>
</meter>
</meters>
---
GET /api/meters/instance.inst1.uptime/samples?start=2010-10-01&end=2010-10-31 HTTP/1.1
Accept: application/xml
HTTP/1.1 200 OK
Content-Type: application/xml
<samples>
<meter id="instance.inst1.uptime" href="/api/meters/instance.ints1.uptime"/>
<sample start="2010-10-16T00:00:00" end="2010-10-16T23:59:59">
<value>3</value>
</sample>
<sample start="2010-10-17T00:00:00" end="2010-10-17T23:59:59">
<value>24</value>
</sample>
<sample start="2010-10-18T00:00:00" end="2010-10-18T23:59:59">
<value>15</value>
</sample>
</samples>
---
GET /api/meters/network.tx/samples?start=2010-10-01&end=2010-10-31 HTTP/1.1
Accept: application/xml
HTTP/1.1 200 OK
Content-Type: application/xml
<samples>
<meter id="network.tx" href="/api/meters/network.tx"/>
<sample start="2010-10-01T00:00:00" end="2010-10-01T23:59:59">
<value>5.2</value>
</sample>
<sample start="2010-10-02T00:00:00" end="2010-10-02T23:59:59">
<value>8.1</value>
</sample>
<sample start="2010-10-03T00:00:00" end="2010-10-03T23:59:59">
<value>0.3</value>
</sample>
...
</samples>
- cost model - defines the costs a user incurs by using various resources
GET /api/costmodels HTTP/1.1
Accept: application/xml
HTTP/1.1 200 OK
Content-Type: application/xml
<cost_models>
<cost_model id="eu-small" href="/api/costmodels/eu-small">
<name>EU Small Instances</name>
<description>Cost model for small EC2 instances in the EU region</description>
<currency id="euro" href="/api/currencies/euro"/>
<link rel="rates" href="/api/costmodels/eu-small/rates"/>
</cost_model>
...
<cost_model id="us-network" href="/api/costmodels/us-network">
<name>US Network I/O</name>
<description>Cost model for network I/O in the US region</description>
<currency id="usd" href="/api/currencies/usd"/>
<link rel="rates" href="/api/costmodels/us-network/rates"/>
</cost_model>
...
</cost_models>
---
GET /api/costmodels/eu-small/rates HTTP/1.1
Accept: application/xml
HTTP/1.1 200 OK
Content-Type: application/xml
<rates>
<rate id="7" href="/api/costmodels/eu-small/rates/7">
<value>9.5</value>
<effective>
<start>2010-01-01T00:00:00</start>
</effective>
<duration id="hourly" href="/api/durations/hourly"/>
</rate>
<rate id="8" href="/api/costmodels/eu-small/rates/8">
<value>10.7</value>
<effective>
<start>2009-01-01T00:00:00</start>
<end>2009-12-31T23:59:59</end>
</effective>
<duration id="hourly" href="/api/durations/hourly"/>
</rate>
...
</rates>
Re: Proposal: Adding billing API support to Deltacloud
Posted by samridh srinath <sa...@gmail.com>.
I think that would be a great place to start Mo.
I'll do that and keep the list updated on my progress so that I can get
everyone's thoughts at every stage. :)
On Wed, Aug 8, 2012 at 8:10 PM, Mo Morsi <mm...@redhat.com> wrote:
> On 08/07/2012 10:24 AM, samridh srinath wrote:
> > Hi David,
> >
> > On Tue, Aug 7, 2012 at 3:37 AM, David Lutterkort <lu...@redhat.com>
> wrote:
> >
> >> Hi Samridh,
> >>
> >> I think that would be a great addition to the DC API. Do you have an
> >> outline what the billing resources should look like, in particular what
> >> data and operations they could support ?
> > I think in terms of operations, as of now clients would only be able to
> GET
> > a bill via the billing API.
> > The bill would be created by the backend cloud provider.
> > I'm still getting around to what data they would support since the kind
> of
> > data returned by ec2 and GoGrid right now are quite different.
> > I'm trying to find a common model that could cover these providers and
> > hopefully all future providers also. It would be helpful if I could get
> > some assistance from existing developers on this end. :)
> >
>
> Sammy, as discussed on IRC it seem that both EC2 and GoGrid make the
> current charges incurred up to this point available via an API call. The
> exact calls vary based on the cloud provider, with gogrid its a simple
> GET request, while with EC2 the user needs to configure a S3 bucket to
> place the billing data in, after which it is available via the normal
> mechanisms to access the bucket (these will be able to be invoked via
> deltacloud correct?). The drivers themselves can take care of these
> implementation details.
>
> Since both describe the amount charged to the account owner as well as
> the justification for those charges (memory usage, ip allocation,
> network traffic, etc) perhaps it makes sense to define a collection
> around 'charges' with data like 'value' and 'reason'. Any other data
> such as time ranges those charges were incurred would be useful as well.
> Perhaps you could flush out a simple xml schema around that and start
> working to define that collection.
>
> It would be nice if these providers exposed a way to retrieve the costs
> of their services via their APIs, but I haven't seen that yet and I'm
> not holding my breath.
>
> Thoughts?
> -Mo
>
Re: Proposal: Adding billing API support to Deltacloud
Posted by Mo Morsi <mm...@redhat.com>.
On 08/07/2012 10:24 AM, samridh srinath wrote:
> Hi David,
>
> On Tue, Aug 7, 2012 at 3:37 AM, David Lutterkort <lu...@redhat.com> wrote:
>
>> Hi Samridh,
>>
>> I think that would be a great addition to the DC API. Do you have an
>> outline what the billing resources should look like, in particular what
>> data and operations they could support ?
> I think in terms of operations, as of now clients would only be able to GET
> a bill via the billing API.
> The bill would be created by the backend cloud provider.
> I'm still getting around to what data they would support since the kind of
> data returned by ec2 and GoGrid right now are quite different.
> I'm trying to find a common model that could cover these providers and
> hopefully all future providers also. It would be helpful if I could get
> some assistance from existing developers on this end. :)
>
Sammy, as discussed on IRC it seem that both EC2 and GoGrid make the
current charges incurred up to this point available via an API call. The
exact calls vary based on the cloud provider, with gogrid its a simple
GET request, while with EC2 the user needs to configure a S3 bucket to
place the billing data in, after which it is available via the normal
mechanisms to access the bucket (these will be able to be invoked via
deltacloud correct?). The drivers themselves can take care of these
implementation details.
Since both describe the amount charged to the account owner as well as
the justification for those charges (memory usage, ip allocation,
network traffic, etc) perhaps it makes sense to define a collection
around 'charges' with data like 'value' and 'reason'. Any other data
such as time ranges those charges were incurred would be useful as well.
Perhaps you could flush out a simple xml schema around that and start
working to define that collection.
It would be nice if these providers exposed a way to retrieve the costs
of their services via their APIs, but I haven't seen that yet and I'm
not holding my breath.
Thoughts?
-Mo
Re: Proposal: Adding billing API support to Deltacloud
Posted by samridh srinath <sa...@gmail.com>.
Hi David,
On Tue, Aug 7, 2012 at 3:37 AM, David Lutterkort <lu...@redhat.com> wrote:
> Hi Samridh,
>
> I think that would be a great addition to the DC API. Do you have an
> outline what the billing resources should look like, in particular what
> data and operations they could support ?
I think in terms of operations, as of now clients would only be able to GET
a bill via the billing API.
The bill would be created by the backend cloud provider.
I'm still getting around to what data they would support since the kind of
data returned by ec2 and GoGrid right now are quite different.
I'm trying to find a common model that could cover these providers and
hopefully all future providers also. It would be helpful if I could get
some assistance from existing developers on this end. :)
>
> David
>
> On Mon, 2012-08-06 at 18:56 +0530, samridh srinath wrote:
> > Hello All,
> >
> > Let me start by introducing myself. I am Samridh, a student developer
> part
> > of Google Summer of Code (GSoC). I have been contributing to the Aeolus
> > project for the better part of the summer, adding API support for
> hardware
> > profiles to the Conductor API. My mentor is Mo Morsi who has been
> assisting
> > me.
> > For what remains of the coding period of GSoC and also after that, Mo
> and I
> > thought that it would be a good idea for me to look into adding billing
> > support to the Deltacloud API. I have also been in conversation with
> > mfojitk and marios in this regard.
> > I will be outlining my research and proposal for this in the remaining
> part
> > of this mail.
> > Currently, Amazon ec2 and GoGrid support programmatic retrieval of
> billing
> > data though in different formats.
> > [1] outlines how to retrieve billing data from GoGrid. It returns data in
> > xml/json/csv via the REST API.
> > [2] outlines how to activate programmatic billing access for an ec2
> > account. Essentially, it involves activating programmatic billing access
> > and then specifying an S3 bucket (either new or an existing bucket)
> where a
> > CSV file containing the usage and billing data will be dumped
> periodically
> > (in the form of an S3 object). The key for retrieving this S3 object is
> in
> > the format "account_number-aws-billing-csv-YYYY-MM.csv".
> > I would like to propose adding billing API support for these two
> providers
> > to start off with and then extending that support as more providers allow
> > for convenient retrieval of billing data.
> > This would essentially require the addition of a new collection, namely
> the
> > "Billing" collection which has a model that has attributes most common
> > among all the providers.
> > The ec2 and gogrid drivers would then be extended to retrieve the billing
> > data from the respective cloud back-ends.
> > The CSV data from the ec2 provider can be converted to xml/json with the
> > help of a rubygem like fastercsv or similar. I am attaching a sample ec2
> > billing CSV file for your reference (Thanks to Mo for that).
> > Any suggestions with respect to the billing model and its attributes are
> > welcome and will definitely help.
> > Also, any other suggestions regarding its implementation are also
> welcome.
> >
> > Looking forward to your feedback and suggestions,
> >
> > Samridh
> > Student Developer,
> > GSoC
> >
> >
> > [1] https://wiki.gogrid.com/wiki/index.php/API:myaccount.billing.get
> > [2]
> >
> http://docs.amazonwebservices.com/awsaccountbilling/latest/about/programaccess.html
>
>
>
>
Re: Proposal: Adding billing API support to Deltacloud
Posted by David Lutterkort <lu...@redhat.com>.
Hi Samridh,
I think that would be a great addition to the DC API. Do you have an
outline what the billing resources should look like, in particular what
data and operations they could support ?
David
On Mon, 2012-08-06 at 18:56 +0530, samridh srinath wrote:
> Hello All,
>
> Let me start by introducing myself. I am Samridh, a student developer part
> of Google Summer of Code (GSoC). I have been contributing to the Aeolus
> project for the better part of the summer, adding API support for hardware
> profiles to the Conductor API. My mentor is Mo Morsi who has been assisting
> me.
> For what remains of the coding period of GSoC and also after that, Mo and I
> thought that it would be a good idea for me to look into adding billing
> support to the Deltacloud API. I have also been in conversation with
> mfojitk and marios in this regard.
> I will be outlining my research and proposal for this in the remaining part
> of this mail.
> Currently, Amazon ec2 and GoGrid support programmatic retrieval of billing
> data though in different formats.
> [1] outlines how to retrieve billing data from GoGrid. It returns data in
> xml/json/csv via the REST API.
> [2] outlines how to activate programmatic billing access for an ec2
> account. Essentially, it involves activating programmatic billing access
> and then specifying an S3 bucket (either new or an existing bucket) where a
> CSV file containing the usage and billing data will be dumped periodically
> (in the form of an S3 object). The key for retrieving this S3 object is in
> the format "account_number-aws-billing-csv-YYYY-MM.csv".
> I would like to propose adding billing API support for these two providers
> to start off with and then extending that support as more providers allow
> for convenient retrieval of billing data.
> This would essentially require the addition of a new collection, namely the
> "Billing" collection which has a model that has attributes most common
> among all the providers.
> The ec2 and gogrid drivers would then be extended to retrieve the billing
> data from the respective cloud back-ends.
> The CSV data from the ec2 provider can be converted to xml/json with the
> help of a rubygem like fastercsv or similar. I am attaching a sample ec2
> billing CSV file for your reference (Thanks to Mo for that).
> Any suggestions with respect to the billing model and its attributes are
> welcome and will definitely help.
> Also, any other suggestions regarding its implementation are also welcome.
>
> Looking forward to your feedback and suggestions,
>
> Samridh
> Student Developer,
> GSoC
>
>
> [1] https://wiki.gogrid.com/wiki/index.php/API:myaccount.billing.get
> [2]
> http://docs.amazonwebservices.com/awsaccountbilling/latest/about/programaccess.html