RE: api test failures - how to resolve?

["Koper, Dies" <di...@fast.au.fujitsu.com>]

Hi Marios,

I've been working on getting the api tests to run on the fgcp.
It's been a good experience as I found some bugs/room for improvement
for the driver.

I have just sent a patch to the ML with changes to both fgcp driver and
tests that address most issues.
Please review.

May I have your feedback on the following:

1. With the fgcp, an instance needs to be started at least once (and
then stopped again) or else you can't create an image of it. So this
test fails. Do you have any suggestion how I can address it? There is no
API to tell whether an instance has been started once.

2. Storage snapshots assume there are snapshots present. I get an error
saying storage_snapshots must have some storage_snapshot elements
inside. I suppose the only way to address this one is to create a volume
and then create a snapshot.

3. Fgcp won't let me create an instance while another resource is being
created. I need to add some waiting logic. I have a patch for that,
which works for fgcp, but I'm not sure it adds inacceptable waiting time
for other drivers. I'll send it separately, as I don't want a NACK for
this one delay an ACK for the first one.

Cheers,
Dies Koper


> -----Original Message-----
> From: Koper, Dies [mailto:diesk@fast.au.fujitsu.com]
> Sent: Friday, 30 November 2012 1:12 AM
> To: dev@deltacloud.apache.org
> Subject: api test failures - how to resolve?
> 
> Hi,
> 
> I've run the API tests on fgcp, i.e. rake test:deltacloud:
> 
> 166 tests, 1340 assertions, 2 failures, 8 errors, 43 skips
> 
> I'd like your suggestions on how to resolve the failures. My goal is
to
> make changes so that a test run like this shows no failures for fgcp
so
> that it can be used for easy regression testing.
> 
>   1) Failure:
> test_0001_must have the "state" element defined for each instance in
> collection(Deltacloud API instances collection)
> [d:/sources/OSS/cloud/deltacloud/tests/delt
> acloud/instances_test.rb:90]:
> Expected [] to not be empty.
> 
> Fgcp driver requires to make an additional backend call for each
> instance in the collection, which caused http timeouts to occur with
my
> (big) contract.
> Therefore, the fgcp driver only includes state when a single instance
is
> retrieved, not for the whole collection.
> I thought that was acceptable, but here it says 'must'?
> 
>   2) Error:
> test_0012_should allow to snapshot running instance if supported by
> provider(Deltacloud API instances collection):
> RestClient::ResourceNotFound: 404 Resource Not Found
> 
> Fgcp only allows snapshots for stopped instances.
> 
> I'll investigate the remaining ones tomorrow.
> 
> Regards,
> Dies Koper
> 
> 

RE: api test failures - how to resolve?

["Koper, Dies" <di...@fast.au.fujitsu.com>]

Hi Marios,

> thanks - can you please check if that one applies for you? I couldn't
> get it to apply - may need a rebase (or otherwise if it works for you
> let me know I can try again...)

I couldn't do a git pull with it so must have forgotten to do a pull
before it, sorry about that.
New one sent 5 min ago.

> discover stuff/skip accordingly. So with this in mind - for this kind
of
> 'fgcp specific' scenario, it may be better to look at adding to the
> white-box tests under /deltacloud/server/tests/drivers... i can't see
> fgcp tests there yet.

Ah you have a whitelist? That's sounds like a nice thing to add to the
readme.
Or better to keep it hidden to get people like me to try harder to get
them to pass? ;)

> gives a clue here. We are talking about creating an image from a
running
> server. In FGCP case, this is not possible. For starters, we shouldn't
> be advertising the 'create_image' action for running instances, when
you
> can't actually create the image from a running instance. If that logic
> were working correctly, this test would just be skipped (i.e.

Of course! The word snapshot confused me (I'm only used to snapshots of
volumes and images of instances), but the key here of course is
"running", and the tests don't start the instance, nor do they check its
state.
I'll look at adding code to check its state.

> fgcp instance running... no create_image...):

Yes, that's covered. The issue is with the test expecting that the
instance is running after create.
I'll look at starting it. In which case my waiting logic in the other
patch makes more sense too :)

> right - either what you suggest here - which is what other tests...
like
> 'instances_test' for example do with the 'setup' stage of creating the
> given resource before running the tests... or rethink the test itself
to
> be more lenient. Either way - its something that should be changed in
> the test - i.e. not in the fgcp driver.

In this case the test was part of code shared by all tests, so making it
more lenient (basically skipping all checks on elements, href etc.)
would affect a lot.
I'll look at doing what instances_test does.

Cheers,
Dies Koper


> -----Original Message-----
> From: marios@redhat.com [mailto:mandreou@redhat.com]
> Sent: Friday, 25 January 2013 7:58 PM
> To: dev@deltacloud.apache.org
> Cc: Koper, Dies
> Subject: Re: api test failures - how to resolve?
> 
> On 25/01/13 09:33, Koper, Dies wrote:
> > Hi Marios,
> >
> > I've been working on getting the api tests to run on the fgcp.
> > It's been a good experience as I found some bugs/room for
improvement
> > for the driver.
> >
> > I have just sent a patch to the ML with changes to both fgcp driver
and
> > tests that address most issues.
> > Please review.
> 
> thanks - can you please check if that one applies for you? I couldn't
> get it to apply - may need a rebase (or otherwise if it works for you
> let me know I can try again...)
> 
> >
> > May I have your feedback on the following:
> >
> > 1. With the fgcp, an instance needs to be started at least once (and
> > then stopped again) or else you can't create an image of it. So this
> > test fails. Do you have any suggestion how I can address it? There
is no
> > API to tell whether an instance has been started once.
> 
> ... can't think of an 'obvious' solution here. You could add some
> waiting logic (as you suggest for 3 below) but I agree with you that
> this may be unacceptable for other drivers; these tests are meant to
me
> 'driver agnostic' - i.e. you just point them at some running
deltacloud
> server with little/no knowledge of what's running there and they
> discover stuff/skip accordingly. So with this in mind - for this kind
of
> 'fgcp specific' scenario, it may be better to look at adding to the
> white-box tests under /deltacloud/server/tests/drivers... i can't see
> fgcp tests there yet.
> 
> I think this may be indicative of a problem with how we advertise
> actions. The name of the test:
> 
> 239 "it 'should allow to snapshot running instance if supported by
> provider' do"
> 
> gives a clue here. We are talking about creating an image from a
running
> server. In FGCP case, this is not possible. For starters, we shouldn't
> be advertising the 'create_image' action for running instances, when
you
> can't actually create the image from a running instance. If that logic
> were working correctly, this test would just be skipped (i.e. fgcp
> instance running... no create_image...):
> 
> 243 skip "no create image support for instance #{@@my_instance_id}"
> unless instance_actions.include?("create_image")$
> 
> This needs more thought - perhaps a JIRA would be a good start.
> 
> >
> > 2. Storage snapshots assume there are snapshots present. I get an
error
> > saying storage_snapshots must have some storage_snapshot elements
> > inside. I suppose the only way to address this one is to create a
volume
> > and then create a snapshot.
> >
> 
> right - either what you suggest here - which is what other tests...
like
> 'instances_test' for example do with the 'setup' stage of creating the
> given resource before running the tests... or rethink the test itself
to
> be more lenient. Either way - its something that should be changed in
> the test - i.e. not in the fgcp driver.
> 
> 
> > 3. Fgcp won't let me create an instance while another resource is
being
> > created. I need to add some waiting logic. I have a patch for that,
> > which works for fgcp, but I'm not sure it adds inacceptable waiting
time
> > for other drivers. I'll send it separately, as I don't want a NACK
for
> > this one delay an ACK for the first one.
> 
> well... ok let's deal with that one when we have the patch to point at
> and discuss,
> 
> 
> thanks, marios
> 
> >
> > Cheers,
> > Dies Koper
> >
> >
> >> -----Original Message-----
> >> From: Koper, Dies [mailto:diesk@fast.au.fujitsu.com]
> >> Sent: Friday, 30 November 2012 1:12 AM
> >> To: dev@deltacloud.apache.org
> >> Subject: api test failures - how to resolve?
> >>
> >> Hi,
> >>
> >> I've run the API tests on fgcp, i.e. rake test:deltacloud:
> >>
> >> 166 tests, 1340 assertions, 2 failures, 8 errors, 43 skips
> >>
> >> I'd like your suggestions on how to resolve the failures. My goal
is
> > to
> >> make changes so that a test run like this shows no failures for
fgcp
> > so
> >> that it can be used for easy regression testing.
> >>
> >>   1) Failure:
> >> test_0001_must have the "state" element defined for each instance
in
> >> collection(Deltacloud API instances collection)
> >> [d:/sources/OSS/cloud/deltacloud/tests/delt
> >> acloud/instances_test.rb:90]:
> >> Expected [] to not be empty.
> >>
> >> Fgcp driver requires to make an additional backend call for each
> >> instance in the collection, which caused http timeouts to occur
with
> > my
> >> (big) contract.
> >> Therefore, the fgcp driver only includes state when a single
instance
> > is
> >> retrieved, not for the whole collection.
> >> I thought that was acceptable, but here it says 'must'?
> >>
> >>   2) Error:
> >> test_0012_should allow to snapshot running instance if supported by
> >> provider(Deltacloud API instances collection):
> >> RestClient::ResourceNotFound: 404 Resource Not Found
> >>
> >> Fgcp only allows snapshots for stopped instances.
> >>
> >> I'll investigate the remaining ones tomorrow.
> >>
> >> Regards,
> >> Dies Koper
> >>
> >>
> >
> >
> 

Re: api test failures - how to resolve?

["marios@redhat.com" <ma...@redhat.com>]

On 25/01/13 09:33, Koper, Dies wrote:
> Hi Marios,
> 
> I've been working on getting the api tests to run on the fgcp.
> It's been a good experience as I found some bugs/room for improvement
> for the driver.
> 
> I have just sent a patch to the ML with changes to both fgcp driver and
> tests that address most issues.
> Please review.

thanks - can you please check if that one applies for you? I couldn't
get it to apply - may need a rebase (or otherwise if it works for you
let me know I can try again...)

> 
> May I have your feedback on the following:
> 
> 1. With the fgcp, an instance needs to be started at least once (and
> then stopped again) or else you can't create an image of it. So this
> test fails. Do you have any suggestion how I can address it? There is no
> API to tell whether an instance has been started once.

... can't think of an 'obvious' solution here. You could add some
waiting logic (as you suggest for 3 below) but I agree with you that
this may be unacceptable for other drivers; these tests are meant to me
'driver agnostic' - i.e. you just point them at some running deltacloud
server with little/no knowledge of what's running there and they
discover stuff/skip accordingly. So with this in mind - for this kind of
'fgcp specific' scenario, it may be better to look at adding to the
white-box tests under /deltacloud/server/tests/drivers... i can't see
fgcp tests there yet.

I think this may be indicative of a problem with how we advertise
actions. The name of the test:

239 "it 'should allow to snapshot running instance if supported by
provider' do"

gives a clue here. We are talking about creating an image from a running
server. In FGCP case, this is not possible. For starters, we shouldn't
be advertising the 'create_image' action for running instances, when you
can't actually create the image from a running instance. If that logic
were working correctly, this test would just be skipped (i.e. fgcp
instance running... no create_image...):

243 skip "no create image support for instance #{@@my_instance_id}"
unless instance_actions.include?("create_image")$

This needs more thought - perhaps a JIRA would be a good start.

> 
> 2. Storage snapshots assume there are snapshots present. I get an error
> saying storage_snapshots must have some storage_snapshot elements
> inside. I suppose the only way to address this one is to create a volume
> and then create a snapshot.
> 

right - either what you suggest here - which is what other tests... like
'instances_test' for example do with the 'setup' stage of creating the
given resource before running the tests... or rethink the test itself to
be more lenient. Either way - its something that should be changed in
the test - i.e. not in the fgcp driver.


> 3. Fgcp won't let me create an instance while another resource is being
> created. I need to add some waiting logic. I have a patch for that,
> which works for fgcp, but I'm not sure it adds inacceptable waiting time
> for other drivers. I'll send it separately, as I don't want a NACK for
> this one delay an ACK for the first one.

well... ok let's deal with that one when we have the patch to point at
and discuss,


thanks, marios

> 
> Cheers,
> Dies Koper
> 
> 
>> -----Original Message-----
>> From: Koper, Dies [mailto:diesk@fast.au.fujitsu.com]
>> Sent: Friday, 30 November 2012 1:12 AM
>> To: dev@deltacloud.apache.org
>> Subject: api test failures - how to resolve?
>>
>> Hi,
>>
>> I've run the API tests on fgcp, i.e. rake test:deltacloud:
>>
>> 166 tests, 1340 assertions, 2 failures, 8 errors, 43 skips
>>
>> I'd like your suggestions on how to resolve the failures. My goal is
> to
>> make changes so that a test run like this shows no failures for fgcp
> so
>> that it can be used for easy regression testing.
>>
>>   1) Failure:
>> test_0001_must have the "state" element defined for each instance in
>> collection(Deltacloud API instances collection)
>> [d:/sources/OSS/cloud/deltacloud/tests/delt
>> acloud/instances_test.rb:90]:
>> Expected [] to not be empty.
>>
>> Fgcp driver requires to make an additional backend call for each
>> instance in the collection, which caused http timeouts to occur with
> my
>> (big) contract.
>> Therefore, the fgcp driver only includes state when a single instance
> is
>> retrieved, not for the whole collection.
>> I thought that was acceptable, but here it says 'must'?
>>
>>   2) Error:
>> test_0012_should allow to snapshot running instance if supported by
>> provider(Deltacloud API instances collection):
>> RestClient::ResourceNotFound: 404 Resource Not Found
>>
>> Fgcp only allows snapshots for stopped instances.
>>
>> I'll investigate the remaining ones tomorrow.
>>
>> Regards,
>> Dies Koper
>>
>>
> 
>