You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@continuum.apache.org by Wendy Smoak <ws...@gmail.com> on 2009/02/11 20:15:40 UTC

Reviving the automated Selenium tests

I'm working on the Selenium tests in src/test/it in the
continuum-webapp-test module.  Right now they're not running at all,
and they depend on some things in maven-shared that are probably left
over from when Continuum and Archiva were both under Maven.  We might
want to bring that over to Continuum, or possibly put it in Redback if
it fits there.

Just fair warning that I'm going to delete a bunch of stuff and do
something simple before building back up to fully automated tests. :)
I'll tag what's there now with today's date to make it easy to bring
back pieces as necessary.

The first step is a dead simple test for the About page, which expects
Continuum to already be running.

I've externalized all the urls, ports, and passwords to a .properties
file so it's easy if you like to run Continuum on a different port or
url.  (For example, running with jetty:run in continuum-webapp, where
it's on port 9090 and doesn't have /continuum in the url.)

>From there we can look at the latest work in the Cargo project to
start/stop containers and deploy the app.  Rumor has it Cargo now
supports multiple data sources for Tomcat.  I'd also like to check out
Don Brown's IT Blast plugin which runs a set of tests against a webapp
in multiple containers.

Comments, suggestions, and volunteers to contribute tests and
configuration are welcome!

-- 
Wendy

Re: Reviving the automated Selenium tests

Posted by Brett Porter <br...@apache.org>.

One thing to keep in mind here - the old tests sucked in some regards  
because they did the whole admin-create-and-login thing on every test.  
I would look at running the tests in a specific order so that a known  
state can be assumed, or look at using something to preload the  
databases for a test case.

Cheers,
Brett

On 12/02/2009, at 3:15 AM, Wendy Smoak wrote:

> I'm working on the Selenium tests in src/test/it in the
> continuum-webapp-test module.  Right now they're not running at all,
> and they depend on some things in maven-shared that are probably left
> over from when Continuum and Archiva were both under Maven.  We might
> want to bring that over to Continuum, or possibly put it in Redback if
> it fits there.
>
> Just fair warning that I'm going to delete a bunch of stuff and do
> something simple before building back up to fully automated tests. :)
> I'll tag what's there now with today's date to make it easy to bring
> back pieces as necessary.
>
> The first step is a dead simple test for the About page, which expects
> Continuum to already be running.
>
> I've externalized all the urls, ports, and passwords to a .properties
> file so it's easy if you like to run Continuum on a different port or
> url.  (For example, running with jetty:run in continuum-webapp, where
> it's on port 9090 and doesn't have /continuum in the url.)
>
> From there we can look at the latest work in the Cargo project to
> start/stop containers and deploy the app.  Rumor has it Cargo now
> supports multiple data sources for Tomcat.  I'd also like to check out
> Don Brown's IT Blast plugin which runs a set of tests against a webapp
> in multiple containers.
>
> Comments, suggestions, and volunteers to contribute tests and
> configuration are welcome!
>
> -- 
> Wendy

--
Brett Porter
brett@apache.org
http://blogs.exist.com/bporter/

Re: Reviving the automated Selenium tests

Posted by Brett Porter <br...@apache.org>.

On 12/02/2009, at 6:16 PM, Emmanuel Venisse wrote:

> If you plan to test Continuum against multiple container/DB, maybe  
> we can
> load some environment on EC2 and tests them with selenium grid. I  
> don't know
> what could be the price for this. Carlos?
>
> WDYT?

I think we should focus on getting them working first, but I'm happy  
to work with you and Carlos to set this up :)

Cheers,
Brett

--
Brett Porter
brett@apache.org
http://blogs.exist.com/bporter/

RE: Reviving the automated Selenium tests

Posted by José Morales Martínez <jm...@gmv.com>.

Thank you very much for the instructions.

I'll start with the option A. Once I was familiar with the test, try researching a little on the line B.

In principle, I will try to generate tests using Java classes.
In my opinion, this option is more reusable and can manage the state of the application.

-----------------------
Jose

-----Mensaje original-----
De: Wendy Smoak [mailto:wsmoak@gmail.com]
Enviado el: vie 13/02/2009 22:57
Para: dev@continuum.apache.org
Asunto: Re: Reviving the automated Selenium tests

On Thu, Feb 12, 2009 at 1:31 PM, José Morales Martínez <jm...@gmv.com> wrote:

> I am interested in this topic.
> I think that this would avoid many of the problems has been the change to strut2.
> How can I contribute? Test will be attached to Jira issues?

Great! Yes, tests can be contributed by opening a JIRA issue and
attaching a patch.

What I'm working on right now is getting the container
start/deploy/stop part automated again.

I've got three parallel efforts going in the continuum-webapp-test module.

(A)
1. in continuum-webapp, run 'mvn jetty:run'
2., in continuum-webapp-test, run 'mvn clean install'

This will use the default pom.xml, and should work. With the webapp
running as the root context on port 9090, you should get a successful
test of the About page. If you want to run tests against another
Continuum instance, edit the url in src/test/resources/it.properties .

Unfortunately, that's the *only* thing that works right now, but if
you're more interested in writing tests than messing with the
infrastructure, start there. :)

Or, if you prefer Selenium IDE, you can record tests with that and
contribute the html files. You'll find some of those tests in
src/test/selenium-ide, and Emmanuel mentioned the idea of
automatically converting them to Java during the build and running
them.

(B)
1. in continuum-webapp-test, run 'mvn clean install -f it-pom.xml'

This should use the Cargo plugin to start Tomcat and deploy the webapp
as /continuum on port 9595. This needs the latest and greatest Cargo,
which you'll probably have to build yourself. You'll see
configuration for the data sources and extra dependency jars in the
pom-it.xml file -- this is new stuff in Cargo.

This one has the most promise. I'm pretty sure it actually worked
once, earlier today, but it's being uncooperative now and I've been
staring at it for too long to see the problem.

(C)
1. edit it.properties and change BASE_URL to
http://localhost:8080/continuum-webapp-1.3.2-SNAPSHOT
2. in continuum-webapp-test, run 'mvn clean install -f pom2.xml'

This should use the JUnit CargoTestSetup class to start Tomcat and
deploy the webapp. It requires a TOMCAT_HOME environment variable set
to the location of a Tomcat 5.5 install. I still need to add the
Resources and figure out how to get the extra jars into the Tomcat
install.

This one isn't ready for general consumption yet. Some things may
still be specific to my environment while I try to get it working.
It's a continuation of some work I did ages ago, and probably isn't
what we'll end up using.

Hope that is enough to get started with, if you have any questions, just ask.

--
Wendy

______________________
Este mensaje, y en su caso, cualquier fichero anexo al mismo,
puede contener informacion clasificada por su emisor como confidencial
en el marco de su Sistema de Gestion de Seguridad de la
Informacion siendo para uso exclusivo del destinatario, quedando
prohibida su divulgacion copia o distribucion a terceros sin la
autorizacion expresa del remitente. Si Vd. ha recibido este mensaje
erroneamente, se ruega lo notifique al remitente y proceda a su borrado.
Gracias por su colaboracion.
______________________
This message including any attachments may contain confidential
information, according to our Information Security Management System,
and intended solely for a specific individual to whom they are addressed.
Any unauthorised copy, disclosure or distribution of this message
is strictly forbidden. If you have received this transmission in error,
please notify the sender immediately and delete it.
______________________

RE: Reviving the automated Selenium tests

Posted by José Morales Martínez <jm...@gmv.com>.

Currently testing with TestNG already been completed, but I am awaiting three points attached new patch:

- CONTINUUM-2083[1]: Waiting a determination on the validation that will be taken.

- CONTINUUM-2145[2]: Waiting for this issue.

- Problem with the execution of the test [3]: I can run the test on my computer, but Wendy has problems (the browser doesn't launch). Someone could try to launch the test to verify if the browser is launched? (Although the tests are not succefully).

EXECUTE: continuum\continuum-webapp-test\ mvn install -f testng-pom.xml

Currently the soruce code repository will not work because isn't created the user 'admin' when the test is initiated. The next patch will solve all problems and finish the test.

[1] http://jira.codehaus.org/browse/CONTINUUM-2083
[2] http://jira.codehaus.org/browse/CONTINUUM-2145
[3] http://jira.codehaus.org/browse/CONTINUUM-2073

-----------------------
Jose

Re: Reviving the automated Selenium tests

Posted by Emmanuel Venisse <em...@gmail.com>.

On Thu, Apr 2, 2009 at 3:20 AM, Brett Porter <br...@apache.org> wrote:

>
> On 02/04/2009, at 12:10 PM, Wendy Smoak wrote:
>
>  Just an update on this, the continuum-webapp-test module now has two poms:
>>
>> "mvn clean install" runs the tests with JUnit and Selenium (with the
>> default pom.xml)
>>
>> "mvn clean install -f testng-pom.xml" runs tests with TestNG and Selenium.
>>
>>
> Is there overlap on these tests, or are they each testing different things?
> If it's the latter, you could run the junit tests using testng and have it
> all in one call.
>
>  We also have some Selenium IDE tests in html format in
>> src/test/selenium-ide.
>>
>> Emmanuel, you once mentioned converting the Selenium IDE tests to Java
>> at build time and running them.  Is that something you've done before
>> or can you point to info on how to do it?  I think that would be
>> interesting to try.
>>
>
> From what I've seen, doing this is likely to get you one big pass/fail of
> whether the suite can run in the given order from scratch (ie you'd need to
> set it up in a clean env.) and makes it harder to fix one test that might
> start failing at some point. Also, the default Java from Selenium IDE tends
> to need a little tweaking.


I'm agree, HTML tests can be view as a big single test.But if we can run
HTML tests files, users will can submit some tests when they have an issue
so we'll can reproduce the error, then we'll can convert them to more
structured java tests


>
> - Brett
>
>

Re: Reviving the automated Selenium tests

Posted by Brett Porter <br...@apache.org>.

On 02/04/2009, at 12:10 PM, Wendy Smoak wrote:

> Just an update on this, the continuum-webapp-test module now has two  
> poms:
>
> "mvn clean install" runs the tests with JUnit and Selenium (with the
> default pom.xml)
>
> "mvn clean install -f testng-pom.xml" runs tests with TestNG and  
> Selenium.
>

Is there overlap on these tests, or are they each testing different  
things? If it's the latter, you could run the junit tests using testng  
and have it all in one call.

> We also have some Selenium IDE tests in html format in src/test/ 
> selenium-ide.
>
> Emmanuel, you once mentioned converting the Selenium IDE tests to Java
> at build time and running them.  Is that something you've done before
> or can you point to info on how to do it?  I think that would be
> interesting to try.

 From what I've seen, doing this is likely to get you one big pass/ 
fail of whether the suite can run in the given order from scratch (ie  
you'd need to set it up in a clean env.) and makes it harder to fix  
one test that might start failing at some point. Also, the default  
Java from Selenium IDE tends to need a little tweaking.

- Brett

Re: Reviving the automated Selenium tests

Posted by Marecor Baclay <mb...@g2ix.com>.

Wendy Smoak wrote:
> Just an update on this, the continuum-webapp-test module now has two poms:
>
> "mvn clean install" runs the tests with JUnit and Selenium (with the
> default pom.xml)
>
> "mvn clean install -f testng-pom.xml" runs tests with TestNG and Selenium.
>
> I'm having trouble with the TestNG tests, though they're working for
> Jose.  The JUnit ones have improved (thanks, Cora!), but still have a
> lot of failures/errors.  Cora, if you have time can you submit a patch
> for the pom that excludes the ones you know are broken?  
Hi Wendy. Ok, I will be working on this next week. Thanks! :)

> Then we could
> get these automated and expect the build to succeed.
>
> We also have some Selenium IDE tests in html format in src/test/selenium-ide.
>
> Emmanuel, you once mentioned converting the Selenium IDE tests to Java
> at build time and running them.  Is that something you've done before
> or can you point to info on how to do it?  I think that would be
> interesting to try.
>
> Thanks,
>

Re: Reviving the automated Selenium tests

Posted by Emmanuel Venisse <em...@gmail.com>.

On Thu, Apr 2, 2009 at 3:10 AM, Wendy Smoak <ws...@gmail.com> wrote:

> Just an update on this, the continuum-webapp-test module now has two poms:
>
> "mvn clean install" runs the tests with JUnit and Selenium (with the
> default pom.xml)
>
> "mvn clean install -f testng-pom.xml" runs tests with TestNG and Selenium.
>
> I'm having trouble with the TestNG tests, though they're working for
> Jose.  The JUnit ones have improved (thanks, Cora!), but still have a
> lot of failures/errors.  Cora, if you have time can you submit a patch
> for the pom that excludes the ones you know are broken?  Then we could
> get these automated and expect the build to succeed.
>
> We also have some Selenium IDE tests in html format in
> src/test/selenium-ide.
>
> Emmanuel, you once mentioned converting the Selenium IDE tests to Java
> at build time and running them.  Is that something you've done before
> or can you point to info on how to do it?  I think that would be
> interesting to try.


I've never done that before before it shoudn't be hard to write the xsl to
convert html files to java.
I've just seen that the selenium mojo have the selenese goal. This goal is
to run HTML files so we don't need to convert HTML files to java.


>
>
> Thanks,
> --
> Wendy
>

Re: Reviving the automated Selenium tests

Posted by Marica Tan <ma...@gmail.com>.

I couldn't checkout continuum trunk rev 767308 from Windows.

It fails with the error:

svn: Can't create directory
'C:\continuum-trunk\continuum-webapp-test\src\test\testng\org\apache\continuum\web\aux'
The directory name is invalid.

'aux' is a reserved word used for DOS devices.

--
Marica

On Sat, Apr 18, 2009 at 5:01 AM, Wendy Smoak <ws...@gmail.com> wrote:

> On Tue, Apr 7, 2009 at 8:08 AM, Marecor Baclay <mb...@g2ix.com> wrote:
>
> > BTW, I modified the pom.xml to use the snapshot version of selenium
> > (1.0-beta-SNAPSHOT) so that we could use Firefox-3.0 in Linux
>
> I think after this change I'm now seeing a _separate_ browser window
> open for the app vs. the Selenium test runner.  Is that configurable?
> I'd rather see the app in the lower frame of a single browser
> instance, the way it used to be.
>
> --
> Wendy
>

Re: Reviving the automated Selenium tests

Posted by Wendy Smoak <ws...@gmail.com>.

On Tue, Apr 7, 2009 at 8:08 AM, Marecor Baclay <mb...@g2ix.com> wrote:

> BTW, I modified the pom.xml to use the snapshot version of selenium
> (1.0-beta-SNAPSHOT) so that we could use Firefox-3.0 in Linux

I think after this change I'm now seeing a _separate_ browser window
open for the app vs. the Selenium test runner.  Is that configurable?
I'd rather see the app in the lower frame of a single browser
instance, the way it used to be.

-- 
Wendy

Re: Reviving the automated Selenium tests

Posted by Marecor Baclay <mb...@g2ix.com>.

Hi Wendy,

I have attached a new patch (CONTINUUM-2073-src-test-it-3.patch) in 
CONTINUUM-2073. I did not exclude the tests that were having errors but 
instead I just commented on the tests that causes errors and repaired 
some of it. As for this patch,

Results :

Tests run: 54, Failures: 0, Errors: 0, Skipped: 0

BTW, I modified the pom.xml to use the snapshot version of selenium 
(1.0-beta-SNAPSHOT) so that we could use Firefox-3.0 in Linux (I haven't 
tried using Firefox-3.0 in windows yet but I guess it will work.) More 
test scripts to add soon. Thanks!

Regards,
Cora

Wendy Smoak wrote:
> Just an update on this, the continuum-webapp-test module now has two poms:
>
> "mvn clean install" runs the tests with JUnit and Selenium (with the
> default pom.xml)
>
> "mvn clean install -f testng-pom.xml" runs tests with TestNG and Selenium.
>
> I'm having trouble with the TestNG tests, though they're working for
> Jose.  The JUnit ones have improved (thanks, Cora!), but still have a
> lot of failures/errors.  Cora, if you have time can you submit a patch
> for the pom that excludes the ones you know are broken?  Then we could
> get these automated and expect the build to succeed.
>
> We also have some Selenium IDE tests in html format in src/test/selenium-ide.
>
> Emmanuel, you once mentioned converting the Selenium IDE tests to Java
> at build time and running them.  Is that something you've done before
> or can you point to info on how to do it?  I think that would be
> interesting to try.
>
> Thanks,
>

Re: Reviving the automated Selenium tests

Posted by Wendy Smoak <ws...@gmail.com>.

Just an update on this, the continuum-webapp-test module now has two poms:

"mvn clean install" runs the tests with JUnit and Selenium (with the
default pom.xml)

"mvn clean install -f testng-pom.xml" runs tests with TestNG and Selenium.

I'm having trouble with the TestNG tests, though they're working for
Jose.  The JUnit ones have improved (thanks, Cora!), but still have a
lot of failures/errors.  Cora, if you have time can you submit a patch
for the pom that excludes the ones you know are broken?  Then we could
get these automated and expect the build to succeed.

We also have some Selenium IDE tests in html format in src/test/selenium-ide.

Emmanuel, you once mentioned converting the Selenium IDE tests to Java
at build time and running them.  Is that something you've done before
or can you point to info on how to do it?  I think that would be
interesting to try.

Thanks,
-- 
Wendy

Re: Reviving the automated Selenium tests

Posted by Marecor Baclay <mb...@exist.com>.

Hi All,

I am repairing the old set of JUnit tests in src/test/it.. I already 
have made it work with Login, Project Group, My Account and Maven 2 
Project tests with some modifications in the old scripts. I will submit 
a patch for this issue (Continuum-2073) later. I ran it using the 
instructions in the ReadMe file found in continuum-webapp-test module. 
Thanks Wendy for the Cargo thing that made the test to run with just 
"mvn clean install" without starting continuum manually.

Cheers,
Cora




Wendy Smoak wrote:
> On Wed, Feb 18, 2009 at 8:39 PM, Wendy Smoak <ws...@gmail.com> wrote:
>
>   
>> Success!  Or, progress, at least.  I think I just re-learned all the
>> lessons that the first person who did this went through. :)
>>     
>
> More progress... the Cargo devs kindly fixed the problem I had with
> the multiple data source support not working with the JNDI mail
> session, so we can now configure the Cargo Maven plugin with
> everything Continuum needs to start and run.
>
> You can see the new config in continuum-webapp-test/pom.xml
>
> That pom goes with the src/test/java/.../AboutTest.java class and
> src/test/resources/it.properties.  I hacked in some conditional code
> in AboutTest to create the admin user and save the general
> configuration, so at long last, 'mvn clean install' works again.
>
> As for the tests themselves, we still have many options open, such as
>  - repairing the old set of JUnit tests in src/test/it
>  - writing new JUnit tests
>  - switching to TestNG (see patch on CONTINUUM-2073, but I need an
> iCLA before adding these)
>  - using html format and converting to Java at build time before
> running the tests
>
> Any further thoughts (or volunteers!) on this?
>
>

Re: Reviving the automated Selenium tests

Posted by Wendy Smoak <ws...@gmail.com>.

On Wed, Feb 18, 2009 at 8:39 PM, Wendy Smoak <ws...@gmail.com> wrote:

> Success!  Or, progress, at least.  I think I just re-learned all the
> lessons that the first person who did this went through. :)

More progress... the Cargo devs kindly fixed the problem I had with
the multiple data source support not working with the JNDI mail
session, so we can now configure the Cargo Maven plugin with
everything Continuum needs to start and run.

You can see the new config in continuum-webapp-test/pom.xml

That pom goes with the src/test/java/.../AboutTest.java class and
src/test/resources/it.properties.  I hacked in some conditional code
in AboutTest to create the admin user and save the general
configuration, so at long last, 'mvn clean install' works again.

As for the tests themselves, we still have many options open, such as
 - repairing the old set of JUnit tests in src/test/it
 - writing new JUnit tests
 - switching to TestNG (see patch on CONTINUUM-2073, but I need an
iCLA before adding these)
 - using html format and converting to Java at build time before
running the tests

Any further thoughts (or volunteers!) on this?

-- 
Wendy

Re: Reviving the automated Selenium tests

Posted by Wendy Smoak <ws...@gmail.com>.

On Fri, Feb 13, 2009 at 2:57 PM, Wendy Smoak <ws...@gmail.com> wrote:

> What I'm working on right now is getting the container
> start/deploy/stop part automated again.

Success!  Or, progress, at least.  I think I just re-learned all the
lessons that the first person who did this went through. :)

If you execute...

   cd continuum-webapp-test
   mvn clean install -f it-pom.xml

... you will see the container start, and then the original tests fail
as they hit the Create Admin User page and never get past it.

The it-pom.xml and src/test/it-resources/it.properties files go with
the code in src/test/it, which is the original set of tests we had.

The pom.xml and src/test/resources/it.properties files go with
src/test/java, which has a single test for the About page, which runs
successfully.  This one requires that you start the webapp yourself --
the easiest way to do that is 'mvn jetty:run' in the continuum-webapp
module.

You can mix and match test code and container start methods by editing
testSourceDirectory and testResources in the poms.

Next on the agenda... either coming up with a new set of tests, or
repairing the old ones.  Any volunteers?

-- 
Wendy

RE: Reviving the automated Selenium tests

Posted by José Morales Martínez <jm...@gmv.com>.

Ok. I am working in this direction. I will attach a patch with some examples. 

NOTE: TestNG license is Apache. It isn't a problem.

-----------------------
Jose

On Mon, Feb 16, 2009 at 10:24 AM, José Morales Martínez
<jm...@gmv.com> wrote:
>
> I noticed that the test uses JUnit, version 3.8.1. As a suggestion, I propose the possibility of using TestNG [1]. In TestNG we can config groups, dependencies inter groups, order, timeOut, ... It also allows the use of annotations.
>
> It's a bit harder to configure, but it has its advantages.

>Sure, let's see how it works.  I'd suggest writing just a few simple
>in JIRA and attach the files, I can add it into the module in another
>parallel directory, and then at some point we can decide which
>direction to go.

-- 
Wendy

______________________
Este mensaje, y en su caso, cualquier fichero anexo al mismo,
 puede contener informacion clasificada por su emisor como confidencial
 en el marco de su Sistema de Gestion de Seguridad de la 
Informacion siendo para uso exclusivo del destinatario, quedando 
prohibida su divulgacion copia o distribucion a terceros sin la 
autorizacion expresa del remitente. Si Vd. ha recibido este mensaje 
 erroneamente, se ruega lo notifique al remitente y proceda a su borrado. 
Gracias por su colaboracion.
______________________
This message including any attachments may contain confidential 
information, according to our Information Security Management System,
 and intended solely for a specific individual to whom they are addressed.
 Any unauthorised copy, disclosure or distribution of this message
 is strictly forbidden. If you have received this transmission in error,
 please notify the sender immediately and delete it.
______________________

Re: Reviving the automated Selenium tests

Posted by Wendy Smoak <ws...@gmail.com>.

On Mon, Feb 16, 2009 at 10:24 AM, José Morales Martínez
<jm...@gmv.com> wrote:
>
> I noticed that the test uses JUnit, version 3.8.1. As a suggestion, I propose the possibility of using TestNG [1]. In TestNG we can config groups, dependencies inter groups, order, timeOut, ... It also allows the use of annotations.
>
> It's a bit harder to configure, but it has its advantages.

Sure, let's see how it works.  I'd suggest writing just a few simple
tests and showing how the configuration is done.  If you open an issue
in JIRA and attach the files, I can add it into the module in another
parallel directory, and then at some point we can decide which
direction to go.

-- 
Wendy

RE: Reviving the automated Selenium tests

Posted by José Morales Martínez <jm...@gmv.com>.

I noticed that the test uses JUnit, version 3.8.1. As a suggestion, I propose the possibility of using TestNG [1]. In TestNG we can config groups, dependencies inter groups, order, timeOut, ... It also allows the use of annotations.

It's a bit harder to configure, but it has its advantages.

WDYT?

[1] http://testng.org