Whirr overhead

[Karel Vervaeke <ka...@outerthought.org>]

I have been using whirr call simple scripts using whirr run-script,
but I have to say it's running way too slow to be usable.

Running parallel-ssh to call a simple script on a 2-node EC2 cluster
takes about 7 seconds.
Running the same script (more or less) takes more than 1min 30 secs.
I know that's comparing I'm apples with comfy chairs: Whirr is making
EC2 calls to check regions,zones, available hosts, ...
but all those EC2 api calls (I have counted 17, see attached log file)
come with a price (well, with time).

Are the EC2 calls really necessary? Maybe we should skip them
alltogether (look in the instances file) or provide caching cache the
EC2 results...

WDYT?
Karel

-- 
Karel Vervaeke
http://outerthought.org/
Open Source Content Applications
Makers of Kauri, Daisy CMS and Lily

Re: Whirr overhead

[Paul Baclace <pa...@gmail.com>]

Allow me to chime in:  absolutely!  There should be no unnecessary EC2 
calls.  I rely on the instances file, although I suppose using spot 
instances could call it into question.  Perhaps spot instances should 
(or are) listed in a different file.


Paul

On 20110926 9:41 , Tom White wrote:
> Karel,
>
> I think using the instances file would be fine for run-script.
>
> Adrian, is it possible to disable some of these API calls in jclouds?
> E.g. DescribeImages is not needed for operations that work on running
> instances.
>
> Cheers,
> Tom
>
> On Mon, Sep 26, 2011 at 9:28 AM, Karel Vervaeke<ka...@outerthought.org>  wrote:
>> I have been using whirr call simple scripts using whirr run-script,
>> but I have to say it's running way too slow to be usable.
>>
>> Running parallel-ssh to call a simple script on a 2-node EC2 cluster
>> takes about 7 seconds.
>> Running the same script (more or less) takes more than 1min 30 secs.
>> I know that's comparing I'm apples with comfy chairs: Whirr is making
>> EC2 calls to check regions,zones, available hosts, ...
>> but all those EC2 api calls (I have counted 17, see attached log file)
>> come with a price (well, with time).
>>
>> Are the EC2 calls really necessary? Maybe we should skip them
>> alltogether (look in the instances file) or provide caching cache the
>> EC2 results...
>>
>> WDYT?
>> Karel
>>
>> --
>> Karel Vervaeke
>> http://outerthought.org/
>> Open Source Content Applications
>> Makers of Kauri, Daisy CMS and Lily
>>

Re: Whirr overhead

[Tom White <to...@cloudera.com>]

Karel,

I think using the instances file would be fine for run-script.

Adrian, is it possible to disable some of these API calls in jclouds?
E.g. DescribeImages is not needed for operations that work on running
instances.

Cheers,
Tom

On Mon, Sep 26, 2011 at 9:28 AM, Karel Vervaeke <ka...@outerthought.org> wrote:
> I have been using whirr call simple scripts using whirr run-script,
> but I have to say it's running way too slow to be usable.
>
> Running parallel-ssh to call a simple script on a 2-node EC2 cluster
> takes about 7 seconds.
> Running the same script (more or less) takes more than 1min 30 secs.
> I know that's comparing I'm apples with comfy chairs: Whirr is making
> EC2 calls to check regions,zones, available hosts, ...
> but all those EC2 api calls (I have counted 17, see attached log file)
> come with a price (well, with time).
>
> Are the EC2 calls really necessary? Maybe we should skip them
> alltogether (look in the instances file) or provide caching cache the
> EC2 results...
>
> WDYT?
> Karel
>
> --
> Karel Vervaeke
> http://outerthought.org/
> Open Source Content Applications
> Makers of Kauri, Daisy CMS and Lily
>