making cimi tests pass on fgcp

["Koper, Dies" <di...@fast.au.fujitsu.com>]

Hi Ronelle, Marios, all

Last week we modified the api tests slightly so they would pass on fgcp.
I would like to do the same with the cimi tests.
Before I complete my patch, I'd like to consult with you to ensure the
direction I'd like to take is acceptable with you.

Unlike the api tests, I appreciate that the cimi tests (at least the
part* ones) are based on documented scenarios that we need to follow. So
I'm trying to come up with a solution that complies with the scenarios.

To get the tests to pass on the fgcp I have to work around the same
restriction with the fgcp endpoint API that affected the api tests: when
you create a resource (machine/volume) or delete one, it does not accept
any other resource creation/deletion requests in that system until the
creation/deletion has completed.

Currently, I'm considering to make the following changes:

1) For tests that create resources and deletion is not part of the test
scenario, perform the deletion in a teardown operation (as is already
done in most cases). The teardown method would loop through the
resources to stop and destroy them. When a destroy operation returns a
405 (Method Not Allowed) or 409 (Conflict), the operation is retried
again and again after a number of seconds.

As the teardown is not part of the scenario, I hope this non-ideal
method is acceptable.

2) For tests that create resources, I'd like to add a checkpoint: that
the resource has actually been created (by sleeping and performing a GET
on it until its state becomes AVAILABLE/ STARTED/ STOPPED. The test then
continues as per the scenario.

I would say this is actually a better implementation of the scenario:
Where e.g. the scenario says the success criteria is "A new Machine
resource is created.", our current test is just checking that the
response of the creation request is 201. There is no check whether the
resource has been created. If it failed during the creation process, our
test would not catch that. With my proposal it would, because we'd
actually be checking that the machine left the CREATING state and
transitioned into a stable success state.

I expect again that added sleeps will not affect performance with the
mock driver. I can imagine the extra check introduced above does incur a
performance impact, depending on the performance of the backend cloud
provider under testing.

What do you think?

Cheers,
Dies Koper

Re: making cimi tests pass on fgcp

[David Lutterkort <lu...@redhat.com>]

On Tue, 2013-02-05 at 15:02 +1100, Koper, Dies wrote:
> So I prefer to apply Band-Aids for a little while longer, so we can at
> least keep testing the cimi front-end with fgcp easily as its coverage
> increases.
> I'd rather look at your solution as a plan B in case the problem will
> not be solved on the fgcp api side and more cimi clients appear.

The problem with applying bandaids to keep the tests working for FGCP is
that we are adapting to non-compliant behavior of the endpoint; and that
also affects the other frontends (DC , EC2)

> > There's actually a good case to be made to add task queue capabilities
> > to Deltacloud for other things, too, and we might get somebody else to
> 
> Yes, I did give it some thinking last month and it's basically a job
> engine we'd have to develop.
> So plan B may be to piggy back on the cimi jobs implementation.
> There is of course the issue that no clients (including DC tests) are
> written to work with that but at least it'd be cimi compliant :-p

This wouldn't have anything to do with CIMI jobs - it would internally
be built on a task queue, but that queue would be invisible to clients.
As an example, right now when a client requests an instance be created,
the driver just turns around and calls the FGCP API. To properly work
around this issue, the driver would throw that instance create request
into a task queue, from which a worker pulls jobs as FGCP is able to
process them. Of course, that means that for every 'GET instance' etc.
request the driver now has to look in the queue and the FGCP backend, as
instances the client thinks of as existing can either be only in the
task queue (and should be reported as pending) or they can be fully
created in FGCP.

> What a queuing system could do, is first try the operation regardless
> (as even if the queue is full, fgcp endpoint will do parameter
> validation) so that in case of invalid parameters or invalid state
> (e.g. trying to delete a machine that is not in the stopped state) the
> client can be notified directly.
> If the endpoint returns an error that the system is locked, add it to
> the queue. The queue periodically retries the operations.
> With machine and volume create operations, we need to return an id. We
> could use the system id + machine name, so at least the machine could
> be programmatically identified later (unless the client creates
> machines with the same name).
> With the public IP address... I suppose a (stateful) job mechanism is the only way?

All of these will require state, and as you point out, some measure of
id generation/translation; but yes, what you describe above is the
correct approach.

David

RE: making cimi tests pass on fgcp

["Koper, Dies" <di...@fast.au.fujitsu.com>]

Hi David,

> Rather than working around this in the tests, we need to take a good
> look at what it takes to beef up the FGCP driver so that DC users will

Actually, I expect this issue (the system blocking) will be picked up as Fujitsu develops its own cimi API and will be addressed in the not too distant future.
So I prefer to apply Band-Aids for a little while longer, so we can at least keep testing the cimi front-end with fgcp easily as its coverage increases.
I'd rather look at your solution as a plan B in case the problem will not be solved on the fgcp api side and more cimi clients appear.

> There's actually a good case to be made to add task queue capabilities
> to Deltacloud for other things, too, and we might get somebody else to

Yes, I did give it some thinking last month and it's basically a job engine we'd have to develop.
So plan B may be to piggy back on the cimi jobs implementation.
There is of course the issue that no clients (including DC tests) are written to work with that but at least it'd be cimi compliant :-p

> Dies, could you write up something that outlines what sort of requests
> need to be buffered and made dependent on each other ?

The scope is the system. Currently all resources are created into a system, they can't be created outside a system.
- machine creation
- machine deletion
- volume creation
- volume deletion
- acquiring a public IP address
- returning a public IP address

What a queuing system could do, is first try the operation regardless (as even if the queue is full, fgcp endpoint will do parameter validation) so that in case of invalid parameters or invalid state (e.g. trying to delete a machine that is not in the stopped state) the client can be notified directly.
If the endpoint returns an error that the system is locked, add it to the queue. The queue periodically retries the operations.
With machine and volume create operations, we need to return an id. We could use the system id + machine name, so at least the machine could be programmatically identified later (unless the client creates machines with the same name).
With the public IP address... I suppose a (stateful) job mechanism is the only way?

Regards,
Dies Koper


> -----Original Message-----
> From: David Lutterkort [mailto:lutter@redhat.com]
> Sent: Tuesday, 5 February 2013 9:35 AM
> To: dev@deltacloud.apache.org
> Subject: Re: making cimi tests pass on fgcp
> 
> On Mon, 2013-02-04 at 16:15 +1100, Koper, Dies wrote:
> > To get the tests to pass on the fgcp I have to work around the same
> > restriction with the fgcp endpoint API that affected the api tests: when
> > you create a resource (machine/volume) or delete one, it does not accept
> > any other resource creation/deletion requests in that system until the
> > creation/deletion has completed.
> 
> I think you have the analysis of the issue spot-on: the issue is that
> the FGCP API does not support the kind of usage that CIMI mandates, and
> that the two other frontends at least implicitly expect.
> 
> That means that no amount of fixing up of tests will actually address
> breakage that will be built into our frontends when using the FGCP
> driver.
> 
> Rather than working around this in the tests, we need to take a good
> look at what it takes to beef up the FGCP driver so that DC users will
> get the behavior they expect. I know that that will require some heavy
> lifting, including making the DC server stateful (at least with the FGCP
> driver) and adding a task queue/daemon, but I'd rather do that than
> force every user of DC to special-case FGCP and be prepared for
> driver-specific behavior.
> 
> There's actually a good case to be made to add task queue capabilities
> to Deltacloud for other things, too, and we might get somebody else to
> work on that. For FGCP, what I am not clear on is the extent of what
> needs to be buffered/sequenced for the API. For example, I'd imagine
> that FGCP instance launching goes into this task queue, and requests get
> buffered there, so that the global locking of FGCP is invisible to the
> user.
> 
> Dies, could you write up something that outlines what sort of requests
> need to be buffered and made dependent on each other ?
> 
> David
> 
> 

Re: making cimi tests pass on fgcp

[David Lutterkort <lu...@redhat.com>]

On Mon, 2013-02-04 at 16:15 +1100, Koper, Dies wrote:
> To get the tests to pass on the fgcp I have to work around the same
> restriction with the fgcp endpoint API that affected the api tests: when
> you create a resource (machine/volume) or delete one, it does not accept
> any other resource creation/deletion requests in that system until the
> creation/deletion has completed.

I think you have the analysis of the issue spot-on: the issue is that
the FGCP API does not support the kind of usage that CIMI mandates, and
that the two other frontends at least implicitly expect.

That means that no amount of fixing up of tests will actually address
breakage that will be built into our frontends when using the FGCP
driver.

Rather than working around this in the tests, we need to take a good
look at what it takes to beef up the FGCP driver so that DC users will
get the behavior they expect. I know that that will require some heavy
lifting, including making the DC server stateful (at least with the FGCP
driver) and adding a task queue/daemon, but I'd rather do that than
force every user of DC to special-case FGCP and be prepared for
driver-specific behavior.

There's actually a good case to be made to add task queue capabilities
to Deltacloud for other things, too, and we might get somebody else to
work on that. For FGCP, what I am not clear on is the extent of what
needs to be buffered/sequenced for the API. For example, I'd imagine
that FGCP instance launching goes into this task queue, and requests get
buffered there, so that the global locking of FGCP is invisible to the
user.

Dies, could you write up something that outlines what sort of requests
need to be buffered and made dependent on each other ?

David

Re: making cimi tests pass on fgcp

[jvlcek <jv...@redhat.com>]

On 02/05/2013 02:29 AM, Koper, Dies wrote:
> Hi Joe,
>
> Thanks for your reply!
>
>> I think this proposal sounds great. The only concern I have is
>> how to address retires that could potentially never succeed.
> There is already retry logic in current tests (both cimi and api) and I
> have been copying what's done there.
> I believe in some that logic was looping endlessly and in others for a
> number of retries (which I had to increase a bit in some cases).
> We can definitely look into capping them all and consistently.
> Maybe we can separate that from this topic as it is an existing issue
> already and will take the patch for this issue longer to develop and
> review.
>
>> I would like to suggest having a retry MAX with a default value
>> that can be customized by someone running the tests. This
>> way the test could be run in a shorter time when the user
>> realizes they may have to do some manual clean-up.
> Retry logic is used in different locations. Are you referring only to
> retry logic in the teardown method?
> Because if you'd decrease the retry max in other locations, it'll just
> mean those tests will fail :)

I was thinking that if special retry logic were to be added to
accommodate the fgcp driver that it should be configurable,
for example with a user configurable MAX retry.



>
>> I would also like to suggest that the test log the state for
>> each iteration of the retry, how many retries were required
>> and the time it took to detect the expected result. This may
> Yes, that is a good point. When I was working on the patch for the api
> tests I added a lot of puts because sometimes I didn't know whether it
> got stuck in an endless loop or whether the tests were still running.
> The problem with my puts was that it made the current output (one line
> test name followed by a '.', 'S' or 'F' status) unreadable so I didn't
> include them in the patch.
> I'd like to hear a suggestion on how to solve that. If we'd make it
> configurable (log level?), would you use it?
> Again, maybe we can separate that from this topic as it is an existing
> issue already and will take the patch for this issue longer to develop
> and review.

Perhaps you could keep an internal counter that could be reported
at the end of the run and optionally produce a spinning pin-wheel
during the run.

>> help us gather some insight into where we may need to try
>> to work to  improve the user experience.
> User experience of?

I was thinking it might help identify possible  bottlenecks
the fgcp driver.


Hope this helps.
   Joe


> It's waiting for (polling) the backend, not much we can do about it in
> Deltacloud.
>
> Cheers,
> Dies Koper
>
>
>> -----Original Message-----
>> From: jvlcek [mailto:jvlcek@redhat.com]
>> Sent: Tuesday, 5 February 2013 1:41 AM
>> To: dev@deltacloud.apache.org
>> Cc: Koper, Dies
>> Subject: Re: making cimi tests pass on fgcp
>>
>> On 02/04/2013 12:15 AM, Koper, Dies wrote:
>>> Hi Ronelle, Marios, all
>>>
>>> Last week we modified the api tests slightly so they would pass on
> fgcp.
>>> I would like to do the same with the cimi tests.
>>> Before I complete my patch, I'd like to consult with you to ensure
> the
>>> direction I'd like to take is acceptable with you.
>>>
>>> Unlike the api tests, I appreciate that the cimi tests (at least the
>>> part* ones) are based on documented scenarios that we need to
> follow. So
>>> I'm trying to come up with a solution that complies with the
> scenarios.
>>> To get the tests to pass on the fgcp I have to work around the same
>>> restriction with the fgcp endpoint API that affected the api tests:
> when
>>> you create a resource (machine/volume) or delete one, it does not
> accept
>>> any other resource creation/deletion requests in that system until
> the
>>> creation/deletion has completed.
>>>
>>> Currently, I'm considering to make the following changes:
>>>
>>> 1) For tests that create resources and deletion is not part of the
> test
>>> scenario, perform the deletion in a teardown operation (as is
> already
>>> done in most cases). The teardown method would loop through the
>>> resources to stop and destroy them. When a destroy operation returns
> a
>>> 405 (Method Not Allowed) or 409 (Conflict), the operation is retried
>>> again and again after a number of seconds.
>>>
>>> As the teardown is not part of the scenario, I hope this non-ideal
>>> method is acceptable.
>>>
>>> 2) For tests that create resources, I'd like to add a checkpoint:
> that
>>> the resource has actually been created (by sleeping and performing a
> GET
>>> on it until its state becomes AVAILABLE/ STARTED/ STOPPED. The test
> then
>>> continues as per the scenario.
>>>
>>> I would say this is actually a better implementation of the
> scenario:
>>> Where e.g. the scenario says the success criteria is "A new Machine
>>> resource is created.", our current test is just checking that the
>>> response of the creation request is 201. There is no check whether
> the
>>> resource has been created. If it failed during the creation process,
> our
>>> test would not catch that. With my proposal it would, because we'd
>>> actually be checking that the machine left the CREATING state and
>>> transitioned into a stable success state.
>>>
>>> I expect again that added sleeps will not affect performance with
> the
>>> mock driver. I can imagine the extra check introduced above does
> incur a
>>> performance impact, depending on the performance of the backend
> cloud
>>> provider under testing.
>>>
>>> What do you think?
>>>
>>> Cheers,
>>> Dies Koper
>>>
>> Dies,
>>
>> I think this proposal sounds great. The only concern I have is
>> how to address retires that could potentially never succeed.
>>
>> I would like to suggest having a retry MAX with a default value
>> that can be customized by someone running the tests. This
>> way the test could be run in a shorter time when the user
>> realizes they may have to do some manual clean-up.
>>
>> I would also like to suggest that the test log the state for
>> each iteration of the retry, how many retries were required
>> and the time it took to detect the expected result. This may
>> help us gather some insight into where we may need to try
>> to work to  improve the user experience.
>>
>> Let me know if I am wrong but I would expect addressing
>> my 2 suggestions would not be overly complex.
>>
>> Hope this helps and thanks,
>>    Joe VLcek
>

RE: making cimi tests pass on fgcp

["Koper, Dies" <di...@fast.au.fujitsu.com>]

Hi Joe,

Thanks for your reply!

> I think this proposal sounds great. The only concern I have is
> how to address retires that could potentially never succeed.

There is already retry logic in current tests (both cimi and api) and I
have been copying what's done there.
I believe in some that logic was looping endlessly and in others for a
number of retries (which I had to increase a bit in some cases).
We can definitely look into capping them all and consistently.
Maybe we can separate that from this topic as it is an existing issue
already and will take the patch for this issue longer to develop and
review.

> I would like to suggest having a retry MAX with a default value
> that can be customized by someone running the tests. This
> way the test could be run in a shorter time when the user
> realizes they may have to do some manual clean-up.

Retry logic is used in different locations. Are you referring only to
retry logic in the teardown method?
Because if you'd decrease the retry max in other locations, it'll just
mean those tests will fail :)

> I would also like to suggest that the test log the state for
> each iteration of the retry, how many retries were required
> and the time it took to detect the expected result. This may

Yes, that is a good point. When I was working on the patch for the api
tests I added a lot of puts because sometimes I didn't know whether it
got stuck in an endless loop or whether the tests were still running.
The problem with my puts was that it made the current output (one line
test name followed by a '.', 'S' or 'F' status) unreadable so I didn't
include them in the patch.
I'd like to hear a suggestion on how to solve that. If we'd make it
configurable (log level?), would you use it?
Again, maybe we can separate that from this topic as it is an existing
issue already and will take the patch for this issue longer to develop
and review.

> help us gather some insight into where we may need to try
> to work to  improve the user experience.

User experience of?
It's waiting for (polling) the backend, not much we can do about it in
Deltacloud.

Cheers,
Dies Koper


> -----Original Message-----
> From: jvlcek [mailto:jvlcek@redhat.com]
> Sent: Tuesday, 5 February 2013 1:41 AM
> To: dev@deltacloud.apache.org
> Cc: Koper, Dies
> Subject: Re: making cimi tests pass on fgcp
> 
> On 02/04/2013 12:15 AM, Koper, Dies wrote:
> > Hi Ronelle, Marios, all
> >
> > Last week we modified the api tests slightly so they would pass on
fgcp.
> > I would like to do the same with the cimi tests.
> > Before I complete my patch, I'd like to consult with you to ensure
the
> > direction I'd like to take is acceptable with you.
> >
> > Unlike the api tests, I appreciate that the cimi tests (at least the
> > part* ones) are based on documented scenarios that we need to
follow. So
> > I'm trying to come up with a solution that complies with the
scenarios.
> >
> > To get the tests to pass on the fgcp I have to work around the same
> > restriction with the fgcp endpoint API that affected the api tests:
when
> > you create a resource (machine/volume) or delete one, it does not
accept
> > any other resource creation/deletion requests in that system until
the
> > creation/deletion has completed.
> >
> > Currently, I'm considering to make the following changes:
> >
> > 1) For tests that create resources and deletion is not part of the
test
> > scenario, perform the deletion in a teardown operation (as is
already
> > done in most cases). The teardown method would loop through the
> > resources to stop and destroy them. When a destroy operation returns
a
> > 405 (Method Not Allowed) or 409 (Conflict), the operation is retried
> > again and again after a number of seconds.
> >
> > As the teardown is not part of the scenario, I hope this non-ideal
> > method is acceptable.
> >
> > 2) For tests that create resources, I'd like to add a checkpoint:
that
> > the resource has actually been created (by sleeping and performing a
GET
> > on it until its state becomes AVAILABLE/ STARTED/ STOPPED. The test
then
> > continues as per the scenario.
> >
> > I would say this is actually a better implementation of the
scenario:
> > Where e.g. the scenario says the success criteria is "A new Machine
> > resource is created.", our current test is just checking that the
> > response of the creation request is 201. There is no check whether
the
> > resource has been created. If it failed during the creation process,
our
> > test would not catch that. With my proposal it would, because we'd
> > actually be checking that the machine left the CREATING state and
> > transitioned into a stable success state.
> >
> > I expect again that added sleeps will not affect performance with
the
> > mock driver. I can imagine the extra check introduced above does
incur a
> > performance impact, depending on the performance of the backend
cloud
> > provider under testing.
> >
> > What do you think?
> >
> > Cheers,
> > Dies Koper
> >
> 
> Dies,
> 
> I think this proposal sounds great. The only concern I have is
> how to address retires that could potentially never succeed.
> 
> I would like to suggest having a retry MAX with a default value
> that can be customized by someone running the tests. This
> way the test could be run in a shorter time when the user
> realizes they may have to do some manual clean-up.
> 
> I would also like to suggest that the test log the state for
> each iteration of the retry, how many retries were required
> and the time it took to detect the expected result. This may
> help us gather some insight into where we may need to try
> to work to  improve the user experience.
> 
> Let me know if I am wrong but I would expect addressing
> my 2 suggestions would not be overly complex.
> 
> Hope this helps and thanks,
>    Joe VLcek

Re: making cimi tests pass on fgcp

[jvlcek <jv...@redhat.com>]

On 02/04/2013 12:15 AM, Koper, Dies wrote:
> Hi Ronelle, Marios, all
>
> Last week we modified the api tests slightly so they would pass on fgcp.
> I would like to do the same with the cimi tests.
> Before I complete my patch, I'd like to consult with you to ensure the
> direction I'd like to take is acceptable with you.
>
> Unlike the api tests, I appreciate that the cimi tests (at least the
> part* ones) are based on documented scenarios that we need to follow. So
> I'm trying to come up with a solution that complies with the scenarios.
>
> To get the tests to pass on the fgcp I have to work around the same
> restriction with the fgcp endpoint API that affected the api tests: when
> you create a resource (machine/volume) or delete one, it does not accept
> any other resource creation/deletion requests in that system until the
> creation/deletion has completed.
>
> Currently, I'm considering to make the following changes:
>
> 1) For tests that create resources and deletion is not part of the test
> scenario, perform the deletion in a teardown operation (as is already
> done in most cases). The teardown method would loop through the
> resources to stop and destroy them. When a destroy operation returns a
> 405 (Method Not Allowed) or 409 (Conflict), the operation is retried
> again and again after a number of seconds.
>
> As the teardown is not part of the scenario, I hope this non-ideal
> method is acceptable.
>
> 2) For tests that create resources, I'd like to add a checkpoint: that
> the resource has actually been created (by sleeping and performing a GET
> on it until its state becomes AVAILABLE/ STARTED/ STOPPED. The test then
> continues as per the scenario.
>
> I would say this is actually a better implementation of the scenario:
> Where e.g. the scenario says the success criteria is "A new Machine
> resource is created.", our current test is just checking that the
> response of the creation request is 201. There is no check whether the
> resource has been created. If it failed during the creation process, our
> test would not catch that. With my proposal it would, because we'd
> actually be checking that the machine left the CREATING state and
> transitioned into a stable success state.
>
> I expect again that added sleeps will not affect performance with the
> mock driver. I can imagine the extra check introduced above does incur a
> performance impact, depending on the performance of the backend cloud
> provider under testing.
>
> What do you think?
>
> Cheers,
> Dies Koper
>

Dies,

I think this proposal sounds great. The only concern I have is
how to address retires that could potentially never succeed.

I would like to suggest having a retry MAX with a default value
that can be customized by someone running the tests. This
way the test could be run in a shorter time when the user
realizes they may have to do some manual clean-up.

I would also like to suggest that the test log the state for
each iteration of the retry, how many retries were required
and the time it took to detect the expected result. This may
help us gather some insight into where we may need to try
to work to  improve the user experience.

Let me know if I am wrong but I would expect addressing
my 2 suggestions would not be overly complex.

Hope this helps and thanks,
   Joe VLcek