rake test in server/

[David Lutterkort <lu...@redhat.com>]

Am I the only one who runs into errors running 'rake test' ? I've got a
bunch of fixes that I'll send out in a minute, but I still get errors
from test:drivers:rhevm - seems like the fixtures need to be rerecorded.

Can others run rake test successfully ?

David

Re: rake test in server/

[jvlcek <jv...@redhat.com>]

On 03/07/2013 02:23 PM, jvlcek wrote:
> On 03/07/2013 12:49 PM, David Lutterkort wrote:
>> On Thu, 2013-03-07 at 09:14 -0500, jvlcek wrote:
>>> On 03/06/2013 08:31 PM, David Lutterkort wrote:
>>>> Am I the only one who runs into errors running 'rake test' ? I've got a
>>>> bunch of fixes that I'll send out in a minute, but I still get errors
>>>> from test:drivers:rhevm - seems like the fixtures need to be rerecorded.
>>>>
>>>> Can others run rake test successfully ?
>>>>
>>>> David
>>>>
>>>>
>>> I had just re-recorded the fixtures for test:drivers:rhevm a few days
>>> ago. Do you have the latest bits?
>> Yes. Part of the problem is that VCR matches on the full URL; the first
>> patch changes it so that it only matches on method, path and query
>> params (though I think RHEV-M doesn't use the latter)
>>
>> We also leak a good bit of information with our recordings. The second
>> (untested) patch tries to scrub user, password, and hostname from the
>> recordings.
>>
>> Finally, I also attach the test failures I get with these changes; it's
>> better than with current HEAD (e884d128). It looks like the failure is
>> genuinely because we need rerecording.
>>
>> BTW, this is with the latest rbovirt (rbovirt-0.0.18)
>>
>> David
>>
>
> The latest bits seem to run fine. See the attached.
>
> I believe the issue is the patch:
> 0001-RHEVM-tests-have-VCR-match-only-on-path-and-query-no.path
> the match_request_on is not needed.
>
> It should use:
> c.default_cassette_options = { :record => :none }
>
>
> I've discussed the leaking information issue with Michal and Marios.
>
> For all future recordings the approach is to use ~/.deltacloud/config to
> do the recordings.
> Then to set the user and pass to fakeuser and fakepassword in the
> recorded fixtures YAML
> files after doing the recording. Which is much easier to do than it sounds.
> This will match the returns from the credentials and driver methods in
> tests/test_helper.rb
>
> For existing recordings the credentials on the cloud providers are to be
> changed so they
> do not match those that had been used during the recording. This does
> leave some UUID
> information for the most part are ephemeral.
>
> Joe V
>

I think the change proposed change in
[PATCH 2/2] RHEVM tests: do not record username/password/hostname with VCR
should align with the returns from credentials and driver methods in
tests/test_helper.r
that is
h.user = "admin"
u.password = "secret"
to
h.user = "fakeuser"
h.password = "fakepassword"

Because it is more possible that admin:secret could be used for live
creds than
fakeuser:fakepassword.

Also could limiting the match_request_on perhaps result in erroneous
matches?

I've posted a "SITE:" change patch to try to document these tests and
how to record them in:http://tracker.deltacloud.org/set/379

Joe V.

Re: rake test in server/

[jvlcek <jv...@redhat.com>]

On 03/07/2013 03:32 PM, David Lutterkort wrote:
> On Thu, 2013-03-07 at 14:23 -0500, jvlcek wrote:
>> The latest bits seem to run fine. See the attached.
> What RUby version do you use ? Can you send me your Gemfile.lock ?

keelhaul % ruby -v
ruby 1.9.3p385 (2013-02-06 revision 39114) [x86_64-linux]


Gemfile.lock attached.

>
>> I believe the issue is the patch:
>> 0001-RHEVM-tests-have-VCR-match-only-on-path-and-query-no.path
>> the match_request_on is not needed.
> No, it is very much needed - without restricting the matching, I get
> even more test failures.

OK using your patches sure but have you tried using a
clean branch == to what's currently in master?

I even pulled a new clone of the repo and the test run fine for me.

>
> It's also not logical that we'd want to match on hostname for RHEV-M. As
> far as the tests are concerned, the host names do not matter.

OK. This is good. If we can limit the matching and still have viable tests
 the whole mess becomes much cleaner... as you elude to with your patches.


>
>> I've discussed the leaking information issue with Michal and Marios.
>>
>> For all future recordings the approach is to use ~/.deltacloud/config to
>> do the recordings.
>> Then to set the user and pass to fakeuser and fakepassword in the
>> recorded fixtures YAML
>> files after doing the recording. Which is much easier to do than it sounds.
> Have you tried out the second patch I sent ? It should do that without
> any sed magic.

I have not but I will and I agree, in reading the patch, that it will
avoid the sed
magic step nicely.. I also suggest using fakeuser:fakepassword vs
admin:secret
since the latter is more realistic and quite possibly actual.
>
>> This will match the returns from the credentials and driver methods in
>> tests/test_helper.rb
> With the changes to match_request_on I sent you, it doesn't matter at
> all what credentials are used during playback - which is what we want.

Sure. And I agree with the "match" approach since you point out the
tests will not be compromised. I had actually tried that but hit the 2
errors
you are so avoided it to help avoid unnecessary re-recordings. I understand
now, and admit, it would have been better to do the rerecording.




>
> It seems to be indeed the case that having a ~/.deltacloud/config during
> playback causes the test failures. I want to get to a point where tests
> pass even if that file exists.

Changes to methods credentials() and driver() in tests/test_helper.rb
could do that. Currently they use for the ~/.deltacloud/config
content if found. Your idea of having an "RE-RECORD" environment
variable is a good one and could be leveraged here too.

>
>> For existing recordings the credentials on the cloud providers are to be
>> changed so they
>> do not match those that had been used during the recording. This does
>> leave some UUID
>> information for the most part are ephemeral.
> I don't care about leaking UUID, but we do have a good amount of real
> user info and hostnames in the recordings right now.
The user info should be handled by changing the creds on the test
systems so they don't match the history in our repo. The hostnames
are an issue but it's not clear to me how much of one if they creds
are not valid.

No doubt moving forward we should avoid recording this info but it's
out there on our repo and the history of the repo will show it. So changing
the creds will help plug the biggest holes.

>
> David
>
>

I'd be glad to take on addressing these issues.

Joe

Re: rake test in server/

[David Lutterkort <lu...@redhat.com>]

On Thu, 2013-03-07 at 14:23 -0500, jvlcek wrote:
> The latest bits seem to run fine. See the attached.

What RUby version do you use ? Can you send me your Gemfile.lock ?

> I believe the issue is the patch:
> 0001-RHEVM-tests-have-VCR-match-only-on-path-and-query-no.path
> the match_request_on is not needed.

No, it is very much needed - without restricting the matching, I get
even more test failures.

It's also not logical that we'd want to match on hostname for RHEV-M. As
far as the tests are concerned, the host names do not matter.

> I've discussed the leaking information issue with Michal and Marios.
> 
> For all future recordings the approach is to use ~/.deltacloud/config to
> do the recordings.
> Then to set the user and pass to fakeuser and fakepassword in the
> recorded fixtures YAML
> files after doing the recording. Which is much easier to do than it sounds.

Have you tried out the second patch I sent ? It should do that without
any sed magic.

> This will match the returns from the credentials and driver methods in
> tests/test_helper.rb

With the changes to match_request_on I sent you, it doesn't matter at
all what credentials are used during playback - which is what we want.

It seems to be indeed the case that having a ~/.deltacloud/config during
playback causes the test failures. I want to get to a point where tests
pass even if that file exists.

> For existing recordings the credentials on the cloud providers are to be
> changed so they
> do not match those that had been used during the recording. This does
> leave some UUID
> information for the most part are ephemeral.

I don't care about leaking UUID, but we do have a good amount of real
user info and hostnames in the recordings right now.

David

Re: rake test in server/

[jvlcek <jv...@redhat.com>]

On 03/07/2013 12:49 PM, David Lutterkort wrote:
> On Thu, 2013-03-07 at 09:14 -0500, jvlcek wrote:
>> On 03/06/2013 08:31 PM, David Lutterkort wrote:
>>> Am I the only one who runs into errors running 'rake test' ? I've got a
>>> bunch of fixes that I'll send out in a minute, but I still get errors
>>> from test:drivers:rhevm - seems like the fixtures need to be rerecorded.
>>>
>>> Can others run rake test successfully ?
>>>
>>> David
>>>
>>>
>> I had just re-recorded the fixtures for test:drivers:rhevm a few days
>> ago. Do you have the latest bits?
> Yes. Part of the problem is that VCR matches on the full URL; the first
> patch changes it so that it only matches on method, path and query
> params (though I think RHEV-M doesn't use the latter)
>
> We also leak a good bit of information with our recordings. The second
> (untested) patch tries to scrub user, password, and hostname from the
> recordings.
>
> Finally, I also attach the test failures I get with these changes; it's
> better than with current HEAD (e884d128). It looks like the failure is
> genuinely because we need rerecording.
>
> BTW, this is with the latest rbovirt (rbovirt-0.0.18)
>
> David
>


The latest bits seem to run fine. See the attached.

I believe the issue is the patch:
0001-RHEVM-tests-have-VCR-match-only-on-path-and-query-no.path
the match_request_on is not needed.

It should use:
c.default_cassette_options = { :record => :none }


I've discussed the leaking information issue with Michal and Marios.

For all future recordings the approach is to use ~/.deltacloud/config to
do the recordings.
Then to set the user and pass to fakeuser and fakepassword in the
recorded fixtures YAML
files after doing the recording. Which is much easier to do than it sounds.
This will match the returns from the credentials and driver methods in
tests/test_helper.rb

For existing recordings the credentials on the cloud providers are to be
changed so they
do not match those that had been used during the recording. This does
leave some UUID
information for the most part are ephemeral.

Joe V

Re: rake test in server/

[David Lutterkort <lu...@redhat.com>]

On Thu, 2013-03-07 at 09:14 -0500, jvlcek wrote:
> On 03/06/2013 08:31 PM, David Lutterkort wrote:
> > Am I the only one who runs into errors running 'rake test' ? I've got a
> > bunch of fixes that I'll send out in a minute, but I still get errors
> > from test:drivers:rhevm - seems like the fixtures need to be rerecorded.
> >
> > Can others run rake test successfully ?
> >
> > David
> >
> >
> I had just re-recorded the fixtures for test:drivers:rhevm a few days
> ago. Do you have the latest bits?

Yes. Part of the problem is that VCR matches on the full URL; the first
patch changes it so that it only matches on method, path and query
params (though I think RHEV-M doesn't use the latter)

We also leak a good bit of information with our recordings. The second
(untested) patch tries to scrub user, password, and hostname from the
recordings.

Finally, I also attach the test failures I get with these changes; it's
better than with current HEAD (e884d128). It looks like the failure is
genuinely because we need rerecording.

BTW, this is with the latest rbovirt (rbovirt-0.0.18)

David

Re: rake test in server/

[jvlcek <jv...@redhat.com>]

On 03/06/2013 08:31 PM, David Lutterkort wrote:
> Am I the only one who runs into errors running 'rake test' ? I've got a
> bunch of fixes that I'll send out in a minute, but I still get errors
> from test:drivers:rhevm - seems like the fixtures need to be rerecorded.
>
> Can others run rake test successfully ?
>
> David
>
>
I had just re-recorded the fixtures for test:drivers:rhevm a few days
ago. Do you have the latest bits?

Joe