[libcloud] pricing update thoughts

[Paul Querna <pa...@querna.org>]

Hi,

I've had this random chat with half a dozen people at this point.

Basically, the idea is to publish a JSON file with a size -> price
mapping for each provider.  This data would be pushed up to an http
server 'somewhere' on the internet.

When provider X changes their prices, we would just push up new json
files, and libcloud copies around the universe would fetch this on
demand.

Thoughts on the basics?

This makes it easier to make a 'long stable' version of libcloud,
because right now we will need to be pushing a new release every
month, because some provider is always changing pricing.

Obviously, we would want to cache the file on the libcloud client
side, and ideally give people a way for people to change the URL,
thinking of a few functions on the top level module, something like:

libcloud.price_cache_update()
libcloud.price_cache_set_url()

"where' to publish this file might be the hardest question, I'd like
to do it on ASF resources, but that might cause some issues on the
infrastructure side, as they have been burned in the past by some java
projects hammering servers for dumb files.

Thanks,

Paul

Re: [libcloud] pricing update thoughts

[Kevin Jackson <fo...@gmail.com>]

Hi,

> Okay, I think leveraging DNS makes a ton of sense.

+1 on this - DNS is very robust and you wouldn't be able to access
cloud services if you can't do a name lookup anyway...

Kev

Re: [libcloud] pricing update thoughts

[Paul Querna <pa...@querna.org>]

On Thu, Jun 3, 2010 at 3:44 AM, Upayavira <uv...@odoko.co.uk> wrote:
> On Wed, 2010-06-02 at 23:39 -0700, Kimbro Staken wrote:
>> Wasn't the goal here to have a very easy way to update prices? How
>> many people actually have access to update DNS records at their
>> company? How about for this project? I guarantee that if you pursue
>> that path you'll get data that's even more stale than having it
>> hardcoded in the source. Also if you want providers to help keep that
>> data current, expecting the person that maintains pricing to have any
>> kind of access to DNS is just not going to be the case and getting a
>> DNS change through the internal process of any modestly sized company
>> can be non-trivial and can take a lot of time. Also if you can't get
>> the ASF to deliver a static file from an HTTP server, what makes you
>> think that getting them to add stuff to DNS will be easy?
>
> Interesting perspective.
>
> Some thoughts/questions:
>
> Who is going to be responsible for maintaining this? Is this going to be
> done on the Apache side or on the provider side?

Committers.  They would update a dict in trunk.  This would get
sync'ed automatically to wherever we store it: DNS or HTTP.

> If it is done on the Apache side, which group of people will need access
> in order to do it? Certainly Paul could edit DNS TXT records, but would
> it be possible/feasible to grant write access to other libcloud
> committers to just that small portion of the DNS setup without giving
> over control to, e.g. libcloud.apache.org?

er, all we would do, is make a python tool that generates a pricing DNS zone.

Then we would for example, delegate the pricing.libcloud.apache.org
DNS name to a separate set of DNS servers, like say
libcloud.jails.apache.org and a few volunteer slaves.

For commiters, they would just commit the updated pricing infomation
to trunk, and we could automate the rest of the chain.

This is relatively similar to what Spam Assassin does with some of
their zones used for spam.

> If we were to go to http, how would we prevent libcloud from downloading
> this data on every method invocation? This has been an issue in the past
> at the ASF, where code pulls from Apache servers every time it is
> invoked (in that case it was pulling a DTD every time it parsed some XML
> that referenced the DTD!!). Libcloud doesn't have any caching mechanisms
> or data storage mechanisms that I am aware of at present, and I'd
> imagine that in many circumstances libcloud doesn't stay resident in
> memory for long enough for simply store it in memory to give sufficient
> caching.

Yes, this is the specific example of why with the ASF Infrastructure
Hat on, I have doubts about hosting it over HTTP, mostly because we
have been burned in the past by projects who don't know about caching,
and were downloading the same freaking XML files hundreds of times per
second from the same IP address.

Re: [libcloud] pricing update thoughts

[Upayavira <uv...@odoko.co.uk>]

On Wed, 2010-06-02 at 23:39 -0700, Kimbro Staken wrote: 
> Wasn't the goal here to have a very easy way to update prices? How
> many people actually have access to update DNS records at their
> company? How about for this project? I guarantee that if you pursue
> that path you'll get data that's even more stale than having it
> hardcoded in the source. Also if you want providers to help keep that
> data current, expecting the person that maintains pricing to have any
> kind of access to DNS is just not going to be the case and getting a
> DNS change through the internal process of any modestly sized company
> can be non-trivial and can take a lot of time. Also if you can't get
> the ASF to deliver a static file from an HTTP server, what makes you
> think that getting them to add stuff to DNS will be easy?

Interesting perspective.

Some thoughts/questions:

Who is going to be responsible for maintaining this? Is this going to be
done on the Apache side or on the provider side?

If it is done on the Apache side, which group of people will need access
in order to do it? Certainly Paul could edit DNS TXT records, but would
it be possible/feasible to grant write access to other libcloud
committers to just that small portion of the DNS setup without giving
over control to, e.g. libcloud.apache.org?

If we were to go to http, how would we prevent libcloud from downloading
this data on every method invocation? This has been an issue in the past
at the ASF, where code pulls from Apache servers every time it is
invoked (in that case it was pulling a DTD every time it parsed some XML
that referenced the DTD!!). Libcloud doesn't have any caching mechanisms
or data storage mechanisms that I am aware of at present, and I'd
imagine that in many circumstances libcloud doesn't stay resident in
memory for long enough for simply store it in memory to give sufficient
caching.

It'll probably turn out that the best place to keep it all is in
Subversion :-P

Upayavira

Re: [libcloud] pricing update thoughts

[Solomon Hykes <so...@dotcloud.com>]

On Wed, Jun 2, 2010 at 11:39 PM, Kimbro Staken <ks...@jumpbox.com> wrote:

> [...] expecting the person that maintains pricing to have any
> kind of access to DNS is just not going to be the case [...]

DNS record would be updated from a single location, every time new
code is pushed to the main repo. So the workflow remains the same:
committers push code with new pricing information.

> Also if you can't get
> the ASF to deliver a static file from an HTTP server, what makes you
> think that getting them to add stuff to DNS will be easy?

Paul's message led me to believe that the ASF needed alternatives to
simply hosting a static file on a public HTTP server. If that's not
the case, obviously there's no need for anything fancy. If, on the
other hand, the kind of traffic generated by a great number of
libcloud clients continuously polling for new pricing information is
actually a problem for ASF's infrastructure, even with http caching
(think many tcp connections), then DNS might be a viable alternative.
I am not familiar with the ASF infrastructure, so I can't answer that.

I think we are among grown-up engineers and all understand that the
simpler the tool, the better.

Best,
Solomon

> On Wed, Jun 2, 2010 at 9:58 PM, Alex Polvi <po...@cloudkick.com> wrote:
>> On Thu, Jun 3, 2010 at 4:39 AM, Jerry Chen <je...@apache.org> wrote:
>>>
>>> On Jun 2, 2010, at 11:34 PM, Solomon Hykes wrote:
>>>
>>>> On Wed, Jun 2, 2010 at 9:24 PM, Dan Di Spaltro <da...@cloudkick.com> wrote:
>>>>> I don't know enough about the DNS protocol, but are there limitations.  This
>>>>> seems like a weird way to do it... Are we just doing it to be novel, or I
>>>>> don't see why a typical webserver serving JSON (pretty well tested) set up
>>>>> isn't good enough.
>>>>
>>>> The 2 things you get for free with DNS are 1. caching and 2. virtually
>>>> unlimited scalability.
>>>>
>>>> I would evaluate how much work and money would be spent on
>>>> implementing a cache system in libcloud and hosting a json file
>>>> reliably. Depending on the answer, using DNS will make sense or not.
>>>> Definitely no point in doing it for the sake of novelty.
>>>
>>> To add to that, this is how I understand this approach in general:
>>>
>>> * Pros for DNS
>>> ** Stable
>>> ** Existing infrastructure
>>> ** Replication/caching is "free"
>>> ** Trust/validity (ownership of domain)
>>> ** Lends to discovery
>>>
>>> * Cons for DNS
>>> ** Propagation delay/limitation
>>> ** Records have a finite length limitation, e.g. SPF needs indirection
>>> ** Different protocol than Provider APIs (DNS v. HTTP)
>>> ** Discovery not usually needed, as a lot of other things are already hard-coded, viz. URL endpoints
>>> ** Non-core library or additional module needed for DNS TXT queries
>>
>> To be fair....
>>
>> * Pros for JSON
>> ** Supported by any language (normally w/out additional dependencies)
>> ** Behaves like the cloud providers themselves do
>> ** Caters to web developers, the likely candidate for people actually
>> building on this stuff
>> ** Human readable with nothing more than curl/browser
>> ** KISS
>>
>> * Cons for JSON
>> ** We have to run a static webserver (wait, that's not hard at all and
>> can just use S3)
>> ** Caching (again, it's a static file, so this is not a problem)
>>
>> I'm not convinced at all that we should complicate things with DNS.
>>
>> -Alex
>>
>> --
>> co-founder, cloudkick.com
>> twitter.com/cloudkick
>> echo V2UgYXJlIGhpcmluZyEK | openssl base64 -d
>>
>

Re: [libcloud] pricing update thoughts

[Kimbro Staken <ks...@jumpbox.com>]

Wasn't the goal here to have a very easy way to update prices? How
many people actually have access to update DNS records at their
company? How about for this project? I guarantee that if you pursue
that path you'll get data that's even more stale than having it
hardcoded in the source. Also if you want providers to help keep that
data current, expecting the person that maintains pricing to have any
kind of access to DNS is just not going to be the case and getting a
DNS change through the internal process of any modestly sized company
can be non-trivial and can take a lot of time. Also if you can't get
the ASF to deliver a static file from an HTTP server, what makes you
think that getting them to add stuff to DNS will be easy?


Kimbro Staken
JumpBox Inc.
http://www.jumpbox.com



On Wed, Jun 2, 2010 at 9:58 PM, Alex Polvi <po...@cloudkick.com> wrote:
> On Thu, Jun 3, 2010 at 4:39 AM, Jerry Chen <je...@apache.org> wrote:
>>
>> On Jun 2, 2010, at 11:34 PM, Solomon Hykes wrote:
>>
>>> On Wed, Jun 2, 2010 at 9:24 PM, Dan Di Spaltro <da...@cloudkick.com> wrote:
>>>> I don't know enough about the DNS protocol, but are there limitations.  This
>>>> seems like a weird way to do it... Are we just doing it to be novel, or I
>>>> don't see why a typical webserver serving JSON (pretty well tested) set up
>>>> isn't good enough.
>>>
>>> The 2 things you get for free with DNS are 1. caching and 2. virtually
>>> unlimited scalability.
>>>
>>> I would evaluate how much work and money would be spent on
>>> implementing a cache system in libcloud and hosting a json file
>>> reliably. Depending on the answer, using DNS will make sense or not.
>>> Definitely no point in doing it for the sake of novelty.
>>
>> To add to that, this is how I understand this approach in general:
>>
>> * Pros for DNS
>> ** Stable
>> ** Existing infrastructure
>> ** Replication/caching is "free"
>> ** Trust/validity (ownership of domain)
>> ** Lends to discovery
>>
>> * Cons for DNS
>> ** Propagation delay/limitation
>> ** Records have a finite length limitation, e.g. SPF needs indirection
>> ** Different protocol than Provider APIs (DNS v. HTTP)
>> ** Discovery not usually needed, as a lot of other things are already hard-coded, viz. URL endpoints
>> ** Non-core library or additional module needed for DNS TXT queries
>
> To be fair....
>
> * Pros for JSON
> ** Supported by any language (normally w/out additional dependencies)
> ** Behaves like the cloud providers themselves do
> ** Caters to web developers, the likely candidate for people actually
> building on this stuff
> ** Human readable with nothing more than curl/browser
> ** KISS
>
> * Cons for JSON
> ** We have to run a static webserver (wait, that's not hard at all and
> can just use S3)
> ** Caching (again, it's a static file, so this is not a problem)
>
> I'm not convinced at all that we should complicate things with DNS.
>
> -Alex
>
> --
> co-founder, cloudkick.com
> twitter.com/cloudkick
> echo V2UgYXJlIGhpcmluZyEK | openssl base64 -d
>

Re: [libcloud] pricing update thoughts

[Alex Polvi <po...@cloudkick.com>]

On Thu, Jun 3, 2010 at 4:39 AM, Jerry Chen <je...@apache.org> wrote:
>
> On Jun 2, 2010, at 11:34 PM, Solomon Hykes wrote:
>
>> On Wed, Jun 2, 2010 at 9:24 PM, Dan Di Spaltro <da...@cloudkick.com> wrote:
>>> I don't know enough about the DNS protocol, but are there limitations.  This
>>> seems like a weird way to do it... Are we just doing it to be novel, or I
>>> don't see why a typical webserver serving JSON (pretty well tested) set up
>>> isn't good enough.
>>
>> The 2 things you get for free with DNS are 1. caching and 2. virtually
>> unlimited scalability.
>>
>> I would evaluate how much work and money would be spent on
>> implementing a cache system in libcloud and hosting a json file
>> reliably. Depending on the answer, using DNS will make sense or not.
>> Definitely no point in doing it for the sake of novelty.
>
> To add to that, this is how I understand this approach in general:
>
> * Pros for DNS
> ** Stable
> ** Existing infrastructure
> ** Replication/caching is "free"
> ** Trust/validity (ownership of domain)
> ** Lends to discovery
>
> * Cons for DNS
> ** Propagation delay/limitation
> ** Records have a finite length limitation, e.g. SPF needs indirection
> ** Different protocol than Provider APIs (DNS v. HTTP)
> ** Discovery not usually needed, as a lot of other things are already hard-coded, viz. URL endpoints
> ** Non-core library or additional module needed for DNS TXT queries

To be fair....

* Pros for JSON
** Supported by any language (normally w/out additional dependencies)
** Behaves like the cloud providers themselves do
** Caters to web developers, the likely candidate for people actually
building on this stuff
** Human readable with nothing more than curl/browser
** KISS

* Cons for JSON
** We have to run a static webserver (wait, that's not hard at all and
can just use S3)
** Caching (again, it's a static file, so this is not a problem)

I'm not convinced at all that we should complicate things with DNS.

-Alex

-- 
co-founder, cloudkick.com
twitter.com/cloudkick
echo V2UgYXJlIGhpcmluZyEK | openssl base64 -d

Re: [libcloud] pricing update thoughts

[Jerry Chen <je...@apache.org>]

On Jun 2, 2010, at 11:34 PM, Solomon Hykes wrote:

> On Wed, Jun 2, 2010 at 9:24 PM, Dan Di Spaltro <da...@cloudkick.com> wrote:
>> I don't know enough about the DNS protocol, but are there limitations.  This
>> seems like a weird way to do it... Are we just doing it to be novel, or I
>> don't see why a typical webserver serving JSON (pretty well tested) set up
>> isn't good enough.
> 
> The 2 things you get for free with DNS are 1. caching and 2. virtually
> unlimited scalability.
> 
> I would evaluate how much work and money would be spent on
> implementing a cache system in libcloud and hosting a json file
> reliably. Depending on the answer, using DNS will make sense or not.
> Definitely no point in doing it for the sake of novelty.

To add to that, this is how I understand this approach in general:

* Pros for DNS
** Stable
** Existing infrastructure
** Replication/caching is "free"
** Trust/validity (ownership of domain)
** Lends to discovery

* Cons for DNS
** Propagation delay/limitation
** Records have a finite length limitation, e.g. SPF needs indirection
** Different protocol than Provider APIs (DNS v. HTTP)
** Discovery not usually needed, as a lot of other things are already hard-coded, viz. URL endpoints
** Non-core library or additional module needed for DNS TXT queries

It definitely would be novel to use DNS, but after writing this all out, perhaps hot potatoes are still the way to go.

Re: [libcloud] pricing update thoughts

[Alex Polvi <po...@cloudkick.com>]

On Thu, Jun 3, 2010 at 5:12 AM, Kevin Jackson <fo...@gmail.com> wrote:
> Hi,
>
> If libcloud is supposed to be a wrapper over various cloud services -
> hosting pricing info on Amazon S3 isn't really eating your own dogfood
> is it?

Maybe instead of adding DNS support, we add storage support to
libcloud, and then seed the file on all of the supported providers.
That'd be dog-fooding it, and pushing the project forward. :)

Or we just post it somewhere random (apache.org, S3, etc)

-Alex

-- 
co-founder, cloudkick.com
twitter.com/cloudkick
echo V2UgYXJlIGhpcmluZyEK | openssl base64 -d

Re: [libcloud] pricing update thoughts

[Kevin Jackson <fo...@gmail.com>]

Hi,

If libcloud is supposed to be a wrapper over various cloud services -
hosting pricing info on Amazon S3 isn't really eating your own dogfood
is it?

Just a thought
Kev

Re: [libcloud] pricing update thoughts

[Solomon Hykes <so...@dotcloud.com>]

Of course, but to quote Paul: "I'd like
to do it on ASF resources".

On Wed, Jun 2, 2010 at 9:39 PM, Dan Di Spaltro <da...@cloudkick.com> wrote:
> S3 comes to mind, basically everything you described.
>
> On Wed, Jun 2, 2010 at 9:34 PM, Solomon Hykes <so...@dotcloud.com> wrote:
>
>> On Wed, Jun 2, 2010 at 9:24 PM, Dan Di Spaltro <da...@cloudkick.com> wrote:
>> > I don't know enough about the DNS protocol, but are there limitations.
>>  This
>> > seems like a weird way to do it... Are we just doing it to be novel, or I
>> > don't see why a typical webserver serving JSON (pretty well tested) set
>> up
>> > isn't good enough.
>>
>> The 2 things you get for free with DNS are 1. caching and 2. virtually
>> unlimited scalability.
>>
>> I would evaluate how much work and money would be spent on
>> implementing a cache system in libcloud and hosting a json file
>> reliably. Depending on the answer, using DNS will make sense or not.
>> Definitely no point in doing it for the sake of novelty.
>>
>
>
>
> --
> Dan Di Spaltro
> cloudkick.com co-founder
> (408) 603-9137
>

Re: [libcloud] pricing update thoughts

[Dan Di Spaltro <da...@cloudkick.com>]

S3 comes to mind, basically everything you described.

On Wed, Jun 2, 2010 at 9:34 PM, Solomon Hykes <so...@dotcloud.com> wrote:

> On Wed, Jun 2, 2010 at 9:24 PM, Dan Di Spaltro <da...@cloudkick.com> wrote:
> > I don't know enough about the DNS protocol, but are there limitations.
>  This
> > seems like a weird way to do it... Are we just doing it to be novel, or I
> > don't see why a typical webserver serving JSON (pretty well tested) set
> up
> > isn't good enough.
>
> The 2 things you get for free with DNS are 1. caching and 2. virtually
> unlimited scalability.
>
> I would evaluate how much work and money would be spent on
> implementing a cache system in libcloud and hosting a json file
> reliably. Depending on the answer, using DNS will make sense or not.
> Definitely no point in doing it for the sake of novelty.
>



-- 
Dan Di Spaltro
cloudkick.com co-founder
(408) 603-9137

Re: [libcloud] pricing update thoughts

[Solomon Hykes <so...@dotcloud.com>]

On Wed, Jun 2, 2010 at 9:24 PM, Dan Di Spaltro <da...@cloudkick.com> wrote:
> I don't know enough about the DNS protocol, but are there limitations.  This
> seems like a weird way to do it... Are we just doing it to be novel, or I
> don't see why a typical webserver serving JSON (pretty well tested) set up
> isn't good enough.

The 2 things you get for free with DNS are 1. caching and 2. virtually
unlimited scalability.

I would evaluate how much work and money would be spent on
implementing a cache system in libcloud and hosting a json file
reliably. Depending on the answer, using DNS will make sense or not.
Definitely no point in doing it for the sake of novelty.

Re: [libcloud] pricing update thoughts

[Dan Di Spaltro <da...@cloudkick.com>]

I don't know enough about the DNS protocol, but are there limitations.  This
seems like a weird way to do it... Are we just doing it to be novel, or I
don't see why a typical webserver serving JSON (pretty well tested) set up
isn't good enough.

On Wed, Jun 2, 2010 at 9:20 PM, Jeremy Orem <je...@gmail.com> wrote:

> It would be nice to, also, have the ability to also point it at a JSON
> file over HTTP:
>
> * other web services could easily pull the data
> * it's easier, in my opinion, to throw some JSON up on a webserver
> than maintain TXT records on a DNS server
> * python has built-in libraries to do this
>
> -Jeremy
>
> On Wed, Jun 2, 2010 at 9:11 PM, Jerry Chen <je...@apache.org> wrote:
> >
> >
> > On Jun 2, 2010, at 11:08 PM, Tom Davis wrote:
> >
> >>> IMHO I think it would be awesome to not have another dependency and
> make a really specialized DNS TXT lookup module.
> >>>
> >>> But I suppose I should back that up with a patch first!
> >>>
> >>> Jerry
> >>
> >> Forgive me, but what is the big issue with dependencies? Why spend
> valuable time writing and debugging a DNS module when dependencies don't
> effect the way 99% of people would install the libcloud package? Even after
> the drawn out anti-zope.interface argument, I still don't get it. If a
> module is well-tested, stable, and available via the cheese shop, there's no
> downside to using it (from my perspective, anyway).
> >
> > Indeed, I took a look at implementing TXT lookups and it's non-trivial.
> >
> > As for Zope, it didn't seem like it was bringing much value.
> >
> > Jerry
>



-- 
Dan Di Spaltro
cloudkick.com co-founder
(408) 603-9137

Re: [libcloud] pricing update thoughts

[Jeremy Orem <je...@gmail.com>]

It would be nice to, also, have the ability to also point it at a JSON
file over HTTP:

* other web services could easily pull the data
* it's easier, in my opinion, to throw some JSON up on a webserver
than maintain TXT records on a DNS server
* python has built-in libraries to do this

-Jeremy

On Wed, Jun 2, 2010 at 9:11 PM, Jerry Chen <je...@apache.org> wrote:
>
>
> On Jun 2, 2010, at 11:08 PM, Tom Davis wrote:
>
>>> IMHO I think it would be awesome to not have another dependency and make a really specialized DNS TXT lookup module.
>>>
>>> But I suppose I should back that up with a patch first!
>>>
>>> Jerry
>>
>> Forgive me, but what is the big issue with dependencies? Why spend valuable time writing and debugging a DNS module when dependencies don't effect the way 99% of people would install the libcloud package? Even after the drawn out anti-zope.interface argument, I still don't get it. If a module is well-tested, stable, and available via the cheese shop, there's no downside to using it (from my perspective, anyway).
>
> Indeed, I took a look at implementing TXT lookups and it's non-trivial.
>
> As for Zope, it didn't seem like it was bringing much value.
>
> Jerry

Re: [libcloud] pricing update thoughts

[Solomon Hykes <so...@dotcloud.com>]

+1. If dnspython does the job, let's use it.

On Wed, Jun 2, 2010 at 9:11 PM, Jerry Chen <je...@apache.org> wrote:
>
>
> On Jun 2, 2010, at 11:08 PM, Tom Davis wrote:
>
>>> IMHO I think it would be awesome to not have another dependency and make a really specialized DNS TXT lookup module.
>>>
>>> But I suppose I should back that up with a patch first!
>>>
>>> Jerry
>>
>> Forgive me, but what is the big issue with dependencies? Why spend valuable time writing and debugging a DNS module when dependencies don't effect the way 99% of people would install the libcloud package? Even after the drawn out anti-zope.interface argument, I still don't get it. If a module is well-tested, stable, and available via the cheese shop, there's no downside to using it (from my perspective, anyway).
>
> Indeed, I took a look at implementing TXT lookups and it's non-trivial.
>
> As for Zope, it didn't seem like it was bringing much value.
>
> Jerry

Re: [libcloud] pricing update thoughts

[Jerry Chen <je...@apache.org>]

On Jun 2, 2010, at 11:08 PM, Tom Davis wrote:

>> IMHO I think it would be awesome to not have another dependency and make a really specialized DNS TXT lookup module.
>> 
>> But I suppose I should back that up with a patch first!
>> 
>> Jerry
> 
> Forgive me, but what is the big issue with dependencies? Why spend valuable time writing and debugging a DNS module when dependencies don't effect the way 99% of people would install the libcloud package? Even after the drawn out anti-zope.interface argument, I still don't get it. If a module is well-tested, stable, and available via the cheese shop, there's no downside to using it (from my perspective, anyway).

Indeed, I took a look at implementing TXT lookups and it's non-trivial.

As for Zope, it didn't seem like it was bringing much value.

Jerry

Re: [libcloud] pricing update thoughts

[Tom Davis <to...@dislocatedday.com>]

On Jun 2, 2010, at 6:40 PM, Jerry Chen <je...@apache.org> wrote:

>
>> On Wed, Jun 2, 2010 at 2:41 PM, Paul Querna <pa...@querna.org> wrote:
>>>
>>> Okay, I think leveraging DNS makes a ton of sense.
>>>
>>> My question is then, how do we have a client from the libcloud side?
>>>
>>> Unless I'm being exceptionally thick headed, I don't see how you can
>>> make requests for txt record types in the standard python library?
>>>
>>> http://docs.python.org/library/socket.html#socket.gethostbyname
>>>
>>> (and the variants gethostbyname_ex and getaddrinfo)
>>>
>>> I am less than enthused if we would need another dependency to  
>>> access
>>> txt records from dns.  There are several possible libraries, PyDNS  
>>> and
>>> dnspython are at the top of google results.
>
>
> On Jun 2, 2010, at 4:51 PM, Ian Kallen wrote:
>
>> FWIW, I previously used dnspython for a homebrewed failover
>> application and elsewhere for dynamic DNS updating, worked great for
>> me in both those cases.
>
> IMHO I think it would be awesome to not have another dependency and  
> make a really specialized DNS TXT lookup module.
>
> But I suppose I should back that up with a patch first!
>
> Jerry

Forgive me, but what is the big issue with dependencies? Why spend  
valuable time writing and debugging a DNS module when dependencies  
don't effect the way 99% of people would install the libcloud package?  
Even after the drawn out anti-zope.interface argument, I still don't  
get it. If a module is well-tested, stable, and available via the  
cheese shop, there's no downside to using it (from my perspective,  
anyway). 

Re: [libcloud] pricing update thoughts

[Jerry Chen <je...@apache.org>]

> On Wed, Jun 2, 2010 at 2:41 PM, Paul Querna <pa...@querna.org> wrote:
>> 
>> Okay, I think leveraging DNS makes a ton of sense.
>> 
>> My question is then, how do we have a client from the libcloud side?
>> 
>> Unless I'm being exceptionally thick headed, I don't see how you can
>> make requests for txt record types in the standard python library?
>> 
>> http://docs.python.org/library/socket.html#socket.gethostbyname
>> 
>> (and the variants gethostbyname_ex and getaddrinfo)
>> 
>> I am less than enthused if we would need another dependency to access
>> txt records from dns.  There are several possible libraries, PyDNS and
>> dnspython are at the top of google results.


On Jun 2, 2010, at 4:51 PM, Ian Kallen wrote:

> FWIW, I previously used dnspython for a homebrewed failover
> application and elsewhere for dynamic DNS updating, worked great for
> me in both those cases.

IMHO I think it would be awesome to not have another dependency and make a really specialized DNS TXT lookup module.

But I suppose I should back that up with a patch first!

Jerry

Re: [libcloud] pricing update thoughts

[Ian Kallen <sp...@gmail.com>]

FWIW, I previously used dnspython for a homebrewed failover
application and elsewhere for dynamic DNS updating, worked great for
me in both those cases.

On Wed, Jun 2, 2010 at 2:41 PM, Paul Querna <pa...@querna.org> wrote:
> On Wed, Jun 2, 2010 at 11:23 AM, Solomon Hykes <so...@dotcloud.com> wrote:
>> On Wed, Jun 2, 2010 at 11:05 AM, Jerry Chen <je...@apache.org> wrote:
>>
>>> It would be awesome if providers would have /pricing.json endpoints so they could update their own prices.
>>>
>>> Even crazier is if people had DNS TXT records with that information.
>>
>> Continuing the craziness... If a provider wishes to maintain prices
>> himself, we could setup CNAMEs so that ec2.prices.libcloud.org points
>> to, say, libcloud-prices.amazonaws.com.
>>
>> But I prefer the idea of keeping everything in the same domain. With a
>> SCM hook you could update the domain with pricing information from the
>> latest revision. When a provider updates pricing information in the
>> code, it's automatically pushed to everyone without the need for an
>> upgrade.
>
>
> Okay, I think leveraging DNS makes a ton of sense.
>
> My question is then, how do we have a client from the libcloud side?
>
> Unless I'm being exceptionally thick headed, I don't see how you can
> make requests for txt record types in the standard python library?
>
> http://docs.python.org/library/socket.html#socket.gethostbyname
>
> (and the variants gethostbyname_ex and getaddrinfo)
>
> I am less than enthused if we would need another dependency to access
> txt records from dns.  There are several possible libraries, PyDNS and
> dnspython are at the top of google results.
>
> Thoughts?
>
> Thanks,
>
> Paul
>



-- 
Ian Kallen
blog: http://www.arachna.com/roller/spidaman
tweetz: http://twitter.com/spidaman
vox: 415.505.5208

Re: [libcloud] pricing update thoughts

[Paul Querna <pa...@querna.org>]

On Wed, Jun 2, 2010 at 11:23 AM, Solomon Hykes <so...@dotcloud.com> wrote:
> On Wed, Jun 2, 2010 at 11:05 AM, Jerry Chen <je...@apache.org> wrote:
>
>> It would be awesome if providers would have /pricing.json endpoints so they could update their own prices.
>>
>> Even crazier is if people had DNS TXT records with that information.
>
> Continuing the craziness... If a provider wishes to maintain prices
> himself, we could setup CNAMEs so that ec2.prices.libcloud.org points
> to, say, libcloud-prices.amazonaws.com.
>
> But I prefer the idea of keeping everything in the same domain. With a
> SCM hook you could update the domain with pricing information from the
> latest revision. When a provider updates pricing information in the
> code, it's automatically pushed to everyone without the need for an
> upgrade.


Okay, I think leveraging DNS makes a ton of sense.

My question is then, how do we have a client from the libcloud side?

Unless I'm being exceptionally thick headed, I don't see how you can
make requests for txt record types in the standard python library?

http://docs.python.org/library/socket.html#socket.gethostbyname

(and the variants gethostbyname_ex and getaddrinfo)

I am less than enthused if we would need another dependency to access
txt records from dns.  There are several possible libraries, PyDNS and
dnspython are at the top of google results.

Thoughts?

Thanks,

Paul

Re: [libcloud] pricing update thoughts

[Solomon Hykes <so...@dotcloud.com>]

On Wed, Jun 2, 2010 at 11:05 AM, Jerry Chen <je...@apache.org> wrote:

> It would be awesome if providers would have /pricing.json endpoints so they could update their own prices.
>
> Even crazier is if people had DNS TXT records with that information.

Continuing the craziness... If a provider wishes to maintain prices
himself, we could setup CNAMEs so that ec2.prices.libcloud.org points
to, say, libcloud-prices.amazonaws.com.

But I prefer the idea of keeping everything in the same domain. With a
SCM hook you could update the domain with pricing information from the
latest revision. When a provider updates pricing information in the
code, it's automatically pushed to everyone without the need for an
upgrade.

Re: [libcloud] pricing update thoughts

[Jerry Chen <je...@apache.org>]

On Jun 2, 2010, at 12:56 PM, Paul Querna wrote:

> Basically, the idea is to publish a JSON file with a size -> price
> mapping for each provider.  This data would be pushed up to an http
> server 'somewhere' on the internet.
> 
> When provider X changes their prices, we would just push up new json
> files, and libcloud copies around the universe would fetch this on
> demand.

I like the idea.

It would be awesome if providers would have /pricing.json endpoints so they could update their own prices.

Even crazier is if people had DNS TXT records with that information.

Looking at my inbox, it appears Solomon is crazy as I am :)

Jerry

Re: [libcloud] pricing update thoughts

[Solomon Hykes <so...@dotcloud.com>]

This might sound silly, but what about DNS? It has caching built-in
and we won't ever have to worry about load.

I could lookup the TXT entry for small.ec2.prices.libcloud.org, for example.

Just my crazy 2 cents.

On Wed, Jun 2, 2010 at 10:56 AM, Paul Querna <pa...@querna.org> wrote:
> Hi,
>
> I've had this random chat with half a dozen people at this point.
>
> Basically, the idea is to publish a JSON file with a size -> price
> mapping for each provider.  This data would be pushed up to an http
> server 'somewhere' on the internet.
>
> When provider X changes their prices, we would just push up new json
> files, and libcloud copies around the universe would fetch this on
> demand.
>
> Thoughts on the basics?
>
> This makes it easier to make a 'long stable' version of libcloud,
> because right now we will need to be pushing a new release every
> month, because some provider is always changing pricing.
>
> Obviously, we would want to cache the file on the libcloud client
> side, and ideally give people a way for people to change the URL,
> thinking of a few functions on the top level module, something like:
>
> libcloud.price_cache_update()
> libcloud.price_cache_set_url()
>
> "where' to publish this file might be the hardest question, I'd like
> to do it on ASF resources, but that might cause some issues on the
> infrastructure side, as they have been burned in the past by some java
> projects hammering servers for dumb files.
>
> Thanks,
>
> Paul
>