You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bloodhound.apache.org by Ryan Ollos <ry...@wandisco.com> on 2012/10/09 11:21:42 UTC
Re: Tests and CI (was: A word on Release 2)
On Fri, Oct 5, 2012 at 10:52 PM, Olemis Lang <ol...@gmail.com> wrote:
> [...]
>
> Besides IMO we should take some time to start writing tests and setup
> CI . This is mainly considering the fact that the number of
> contributors has increased recently and more concurrent changes
> against a more complex implementation means a higher probability to
> e.g. introduce regressions.
>
Additionally, it would be very nice to have some functional/acceptance
tests for plugins that we pull from a third-party source (primarily
trac-hacks, I imagine). The acceptance test suite might be pretty small to
start with, but having the framework in place at least allows regressions
to be dealt with by putting a test in place. Some recent work was leading
me to consider this - a little off topic, but I'll circle back around here
in a second ;)
Over the past 2-3 days I've been doing some work on the
MasterTicketsPlugin, with the idea that this could possibly be included in
Bloodhound to support a certain type of ticket dependencies. At this stage
I was just study the code and thinking about the features; my next step
will be to raise the issue on the mailing list. I wrote some unit tests,
fixed some bugs and refactored a bunch of code. I feel like it is getting
better, but it would sure be nice to have some functional tests at this
stage, and even more so before pulling a plugin like this into the
Bloodhound project.
So finally, my question. The Trac project uses Twill for functional
testing. I see some tests in bhdashboard, but they appear to be unit tests
from what I can tell. Has there been any discussion or exploration of what
tool(s) to use for functional testing in Bloodhound? I have a very small
amount of experience writing tests in Twill, so I have no idea if that is a
relatively good suite to use. Is there anyone with more experience that can
provide some suggestions? I was planning to try setting up some tests in
Twill for MasterTicketsPlugin and talk about that more when I raise that
issue for discussion soon, but if someone has a better suggestion I would
give it a try.
Re: Tests and CI (was: A word on Release 2)
Posted by Olemis Lang <ol...@gmail.com>.
On 10/10/12, Ryan Ollos <ry...@wandisco.com> wrote:
> On Tue, Oct 9, 2012 at 8:16 PM, Olemis Lang <ol...@gmail.com> wrote:
>
>> [...]
>>
>> What do you mean exactly ? Something like using Gherkin (e.g. Lettuce)
>> to describe user stories ?
>>
>
> That's not what I had in mind, but I have a general idea of what you are
> talking about and also think that is a good idea to pursue at some point.
>
[...]
... or maybe not , that's what we need to figure out
;)
I mentioned Gherkin because I read «acceptance tests» in your previous
message . Most of the time this is associated with Ghenkin-like
framework , robot framework , PyFIT , or similar approaches (there's a
category in PythonTestingTaxonomy page if you want to know more about
this ;) . They are a bit different to what we'll need for functional
testing . That's why I asked . I just wanted to ensure we were using
the same vocabulary to refer to the same thing. Fortunately I
understand what you mean now
:)
>>
>> If you mention that after reviewing Trac tests , yes ... Trac
>> unitest-style is a PITA ... that's why I wrote my own framework in the
>> end
>> ;)
>
>
> Well, I mention after having written some tests when submitting patches to
> the Trac core.
ok , jftr , I even know people who don't write tests for their plugins
just because it's a PITA . But anyway ...
I guess it wasn't all that bad, but I suspect there is
> framework that allows more concise and readable tests. I'm not discounting
> your way of doing it because I haven't taken a close look at what you've
> done with Twill yet,
... I'll adapt and use whatever you decide . that was jftr .
;)
--
Regards,
Olemis.
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
Featured article:
Re: Tests and CI (was: A word on Release 2)
Posted by Gary Martin <ga...@wandisco.com>.
On 11/10/12 03:41, Branko Čibej wrote:
> On 11.10.2012 02:12, Gary Martin wrote:
>> On 10/10/12 18:26, Peter Koželj wrote:
>>> I admit it, some of my fears could just be the cause of the fact that
>>> I am
>>> not familiar with Trac plugin architecture and the Trac plugin
>>> ecosystem as
>>> a whole yet.
>>>
>>> Just a few use cases for quick "does it float" test:
>>> 1. Let's say we adopt some plugins for multiproducts, ticket relations,
>>> data import/export and so on...
>>> 2. Then we want to add user (admin) defined ticket relation per product
>>> 3. And then we want to support custom ticket workflows per product
>>> and per
>>> ticket type
>>> 4. And now we want the ticket import/export from/to Jira to work across
>>> Trac and all other plugins (including ticket relations, multiproducts
>>> and
>>> custom per product workflows plugins)
>>> 5. Add white labeling across Trac and all other plugins
>>> 6. Implement new search query language that will understand tickets,
>>> products and relations
>>> 7. Add REST API's for all of the above
>>> 8. ...
>>>
>>> Now, all this plugins (products, relations, import, export, search,
>>> white
>>> labeling, new search...) will either have to know filthy little details
>>> about each other (complex inter-plugin dependency) or some super heavy
>>> shifting in the plugin interface will be necessary. And that super heavy
>>> shifting will have to be pushed back to Trac (if it wants it) and
>>> only then
>>> to these plugins or create new Bloodhound plugin API on top of Trac
>>> plugin
>>> API and then request plugin authors to support it.
>>>
>>> Am I missing something?
>> I am not sure that I see the same difficulties. It seems a pretty
>> normal thing for a program to be divided into components in such a way
>> that keeps filthy details hidden. All we are doing here is allowing
>> the set of components to be distributed between core trac, bloodhound
>> plugins and external plugins. This is achieved through Trac and
>> plugins defining interfaces, with components then being able to
>> implement the interfaces and/or iterate through components that
>> provide the interface.
>>
>> So far I have found little reason to put any "super heavy shifting"
>> into trac itself as they appear to have provided enough interfaces to
>> allow for some fairly complex subverting of standard trac processing.
>> The namespacing for multiproducts is a case in point where, so far, it
>> appears to be achievable outside of trac. If it is not possible to do
>> all that we want to with, say, per-product configuration or workflow,
>> it might be that we will need to introduce a new interface to Trac to
>> allow us to take over processing at the appropriate time but I am
>> hoping that this will not be necessary.
> The point Peter is trying to make is this: if we take as an example the
> multiproduct plugin:
>
> This plugin has to change the trac database schema (or rather, add a
> more-or-less independent namespace mapping to the existing schema).
> Consequentially, plugins must be allowed to either update the core
> schema, or create their own additional schema. Whichever it is affects
>
> * database schema upgrades (from any older version to any newer version)
> o either each plugin has to have its own schema upgrade mechanism
> -- a nightmare, or
> o the plugin API must define unified schema management protocols,
> wich /all/ plugins that fiddle with the schema have to conform to
Whether it is a nightmare or not, the responsibility for schema upgrade
is on the shoulders of the plugin. If we want to make this a more
uniform experience, and if there is no solution within trac, we would
probably have to create a plugin that would provide these services and
hope that the plugins we want to use will adopt this as a dependency. Of
course, it would be essential that this plugin stays compatible with
trac and it would effectively need to remain distributable as a
separable entity from the rest of bloodhound unless we can convince
upstream trac of the need for this early on.
> * data export/import
> o as above, either each plugin handles its own export/import (and
> incidentally defines its own export format), which implies that
> anyone who wants to write an exporter from another issue
> tracking system to Bloodhound's export format has to know
> exactly which plugins are in fact part of core Bloodhound; or,
> o the plugin API must define a common event streaming protocol
> that can be converted to a generic export format; which does not
> seem trivial to me.
>
> As another example, a "new search" plugin would conceivably allow
> defining event notification triggers based on the results of a custom
> query having changed. Any export format has to have a way to represent
> this (if only because JIRA has this feature and you'd eventually want to
> be able to import these definitions if BH achieves feature parity in
> that respect).
>
> Almost every new feature that Bloodhound adopts and implements via a
> plugin affects with the goal of creating a common export format, a
> concept that trac does not have and therefore isn't likely to be
> supported in the plugin API.
For me your argument continues to suggest that this is a problem for all
plugins which it is not.
The only problem is with external plugins as they may not want to
implement the interfaces that we want them to, or indeed implement them
correctly. We have to evaluate external plugins to see what problems
there are and decide what matters. I don't want us to cultivate a not
invented here attitude whilst we are able to benefit from the work of
others and can attempt to provide reciprocal benefits.
Cheers,
Gary
Re: Tests and CI (was: A word on Release 2)
Posted by Branko Čibej <br...@wandisco.com>.
On 11.10.2012 02:12, Gary Martin wrote:
> On 10/10/12 18:26, Peter Koželj wrote:
>> I admit it, some of my fears could just be the cause of the fact that
>> I am
>> not familiar with Trac plugin architecture and the Trac plugin
>> ecosystem as
>> a whole yet.
>>
>> Just a few use cases for quick "does it float" test:
>> 1. Let's say we adopt some plugins for multiproducts, ticket relations,
>> data import/export and so on...
>> 2. Then we want to add user (admin) defined ticket relation per product
>> 3. And then we want to support custom ticket workflows per product
>> and per
>> ticket type
>> 4. And now we want the ticket import/export from/to Jira to work across
>> Trac and all other plugins (including ticket relations, multiproducts
>> and
>> custom per product workflows plugins)
>> 5. Add white labeling across Trac and all other plugins
>> 6. Implement new search query language that will understand tickets,
>> products and relations
>> 7. Add REST API's for all of the above
>> 8. ...
>>
>> Now, all this plugins (products, relations, import, export, search,
>> white
>> labeling, new search...) will either have to know filthy little details
>> about each other (complex inter-plugin dependency) or some super heavy
>> shifting in the plugin interface will be necessary. And that super heavy
>> shifting will have to be pushed back to Trac (if it wants it) and
>> only then
>> to these plugins or create new Bloodhound plugin API on top of Trac
>> plugin
>> API and then request plugin authors to support it.
>>
>> Am I missing something?
>
> I am not sure that I see the same difficulties. It seems a pretty
> normal thing for a program to be divided into components in such a way
> that keeps filthy details hidden. All we are doing here is allowing
> the set of components to be distributed between core trac, bloodhound
> plugins and external plugins. This is achieved through Trac and
> plugins defining interfaces, with components then being able to
> implement the interfaces and/or iterate through components that
> provide the interface.
>
> So far I have found little reason to put any "super heavy shifting"
> into trac itself as they appear to have provided enough interfaces to
> allow for some fairly complex subverting of standard trac processing.
> The namespacing for multiproducts is a case in point where, so far, it
> appears to be achievable outside of trac. If it is not possible to do
> all that we want to with, say, per-product configuration or workflow,
> it might be that we will need to introduce a new interface to Trac to
> allow us to take over processing at the appropriate time but I am
> hoping that this will not be necessary.
The point Peter is trying to make is this: if we take as an example the
multiproduct plugin:
This plugin has to change the trac database schema (or rather, add a
more-or-less independent namespace mapping to the existing schema).
Consequentially, plugins must be allowed to either update the core
schema, or create their own additional schema. Whichever it is affects
* database schema upgrades (from any older version to any newer version)
o either each plugin has to have its own schema upgrade mechanism
-- a nightmare, or
o the plugin API must define unified schema management protocols,
wich /all/ plugins that fiddle with the schema have to conform to
* data export/import
o as above, either each plugin handles its own export/import (and
incidentally defines its own export format), which implies that
anyone who wants to write an exporter from another issue
tracking system to Bloodhound's export format has to know
exactly which plugins are in fact part of core Bloodhound; or,
o the plugin API must define a common event streaming protocol
that can be converted to a generic export format; which does not
seem trivial to me.
As another example, a "new search" plugin would conceivably allow
defining event notification triggers based on the results of a custom
query having changed. Any export format has to have a way to represent
this (if only because JIRA has this feature and you'd eventually want to
be able to import these definitions if BH achieves feature parity in
that respect).
Almost every new feature that Bloodhound adopts and implements via a
plugin affects with the goal of creating a common export format, a
concept that trac does not have and therefore isn't likely to be
supported in the plugin API.
-- Brane
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Re: Tests and CI (was: A word on Release 2)
Posted by Olemis Lang <ol...@gmail.com>.
On 10/10/12, Gary Martin <ga...@wandisco.com> wrote:
> On 10/10/12 18:26, Peter Koželj wrote:
>>
[...]
>>
>> Am I missing something?
>
[...]
>
> The
> namespacing for multiproducts is a case in point where, so far, it
> appears to be achievable outside of trac.
+1
> If it is not possible to do
> all that we want to with, say, per-product configuration or workflow,
jftr ... afaics , it will be . I hope we all can share the same vision
once I share aforementioned proposal .
> it
> might be that we will need to introduce a new interface to Trac to allow
> us to take over processing at the appropriate time but I am hoping that
> this will not be necessary.
>
/me too ...
;)
--
Regards,
Olemis.
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
Featured article:
Re: Tests and CI (was: A word on Release 2)
Posted by Gary Martin <ga...@wandisco.com>.
On 10/10/12 18:26, Peter Koželj wrote:
> I admit it, some of my fears could just be the cause of the fact that I am
> not familiar with Trac plugin architecture and the Trac plugin ecosystem as
> a whole yet.
>
> Just a few use cases for quick "does it float" test:
> 1. Let's say we adopt some plugins for multiproducts, ticket relations,
> data import/export and so on...
> 2. Then we want to add user (admin) defined ticket relation per product
> 3. And then we want to support custom ticket workflows per product and per
> ticket type
> 4. And now we want the ticket import/export from/to Jira to work across
> Trac and all other plugins (including ticket relations, multiproducts and
> custom per product workflows plugins)
> 5. Add white labeling across Trac and all other plugins
> 6. Implement new search query language that will understand tickets,
> products and relations
> 7. Add REST API's for all of the above
> 8. ...
>
> Now, all this plugins (products, relations, import, export, search, white
> labeling, new search...) will either have to know filthy little details
> about each other (complex inter-plugin dependency) or some super heavy
> shifting in the plugin interface will be necessary. And that super heavy
> shifting will have to be pushed back to Trac (if it wants it) and only then
> to these plugins or create new Bloodhound plugin API on top of Trac plugin
> API and then request plugin authors to support it.
>
> Am I missing something?
I am not sure that I see the same difficulties. It seems a pretty normal
thing for a program to be divided into components in such a way that
keeps filthy details hidden. All we are doing here is allowing the set
of components to be distributed between core trac, bloodhound plugins
and external plugins. This is achieved through Trac and plugins defining
interfaces, with components then being able to implement the interfaces
and/or iterate through components that provide the interface.
So far I have found little reason to put any "super heavy shifting" into
trac itself as they appear to have provided enough interfaces to allow
for some fairly complex subverting of standard trac processing. The
namespacing for multiproducts is a case in point where, so far, it
appears to be achievable outside of trac. If it is not possible to do
all that we want to with, say, per-product configuration or workflow, it
might be that we will need to introduce a new interface to Trac to allow
us to take over processing at the appropriate time but I am hoping that
this will not be necessary.
Cheers,
Gary
Re: Tests and CI (was: A word on Release 2)
Posted by Peter Koželj <pe...@digiverse.si>.
I admit it, some of my fears could just be the cause of the fact that I am
not familiar with Trac plugin architecture and the Trac plugin ecosystem as
a whole yet.
Just a few use cases for quick "does it float" test:
1. Let's say we adopt some plugins for multiproducts, ticket relations,
data import/export and so on...
2. Then we want to add user (admin) defined ticket relation per product
3. And then we want to support custom ticket workflows per product and per
ticket type
4. And now we want the ticket import/export from/to Jira to work across
Trac and all other plugins (including ticket relations, multiproducts and
custom per product workflows plugins)
5. Add white labeling across Trac and all other plugins
6. Implement new search query language that will understand tickets,
products and relations
7. Add REST API's for all of the above
8. ...
Now, all this plugins (products, relations, import, export, search, white
labeling, new search...) will either have to know filthy little details
about each other (complex inter-plugin dependency) or some super heavy
shifting in the plugin interface will be necessary. And that super heavy
shifting will have to be pushed back to Trac (if it wants it) and only then
to these plugins or create new Bloodhound plugin API on top of Trac plugin
API and then request plugin authors to support it.
Am I missing something?
Peter
On Wed, Oct 10, 2012 at 5:22 PM, Olemis Lang <ol...@gmail.com> wrote:
> On 10/10/12, Ryan Ollos <ry...@wandisco.com> wrote:
> > On Wed, Oct 10, 2012 at 1:27 AM, Peter Koželj <pe...@digiverse.si>
> wrote:
> >
> >> This my get a bit off topic...
> >>
> >> I agree that testing is important, and we probably agree that testing is
> >> hard.
> >> In fact a good comprehensive test of a feature can take more effort to
> >> write and maintain that the actual code that implements that feature.
>
> +1
>
> >> So
> >> if
> >> plugins do not provide it's own tests and we need to write and maintain
> >> the
> >> tests for them, the benefit of using them get's smaller.
> >>
>
> Well . The real benefit is that we'll be able to sleep at night
> knowing that if some change is introduced (Bloodhound core ,
> trac-hacks , plugins ...) and causes some trouble then we are just one
> day away to figure it out ... especially if CI is installed .
> ;)
>
> For me the benefit exists , but we certainly need to assess the
> additional workload involved and write tests when necessary .
>
> >> I still need to come to terms
> >> with the idea that such a core functionality as ticket relations or
> multi
> >> products needs to be a plugin or even worse an external plugin! With all
> >> the features I would like to see in a issue tracker, I just can not
> >> imagine
> >> how this will work without complex inter-plugin dependencies,
> >> extremely cooperational plugin authors
>
> Notice that we are plugin maintainers of many plugins @ t-h.o , so no big
> deal
>
> <joke>
> ... and I swear I'll cooperate with Ryan as long as at any given time
> the number of test cases he writes is smaller than those I wrote ...
> </joke>
>
> >> and last but no least, all the
> >> time
> >> in the world :)
> >>
>
> Oh no ! No way . Exactly the same time as we were doing it using a
> plugin delivered by Bloodhound itself . In case plugin maintainers are
> not responsive and not willing to grant maintainership ... is not a
> chaos ... we can fork the project if there's a merit
>
> >
> > I agree that you may not want the critical Bloodhound functionality to
> live
> > externally, and to rely on developers not involved with the Bloodhound
> > project to maintain it. It might be better to have any critical
> > functionality be written as a plugin that lives in the Bloodhound
>
> Indeed , we *NEED* plugin authors . There's no community without them
> ;)
>
> > The idea of the functionality not being a plugin is more
> > difficult for me to imagine.
> >
>
> definitely sure . Trac itself , I mean the core (tickets , vcs , ...)
> maybe packaged as separate plugins as well .
>
> > Not writing it as a plugin means you are
> > just modifying existing Components in Trac (such as trac.ticket.*) to
> > provide your functionality, and this would make merging from the Trac
> > mainline more difficult, and in which case the tests potentially become
> > even more important.
> >
>
> Interesting . Not necessarily harder . That depends . Indeed we have
> overridden TicketModule but in a clever way (Gary is guilty for that
> ;) so as make merge process easier .
>
> > A feature like ticket dependencies could be written as a plugin (i.e.
> > Component) that is maintained within Bloodhound and is always enabled in
> > the Bloodhound core, which might get around some of your concerns about
> > relying on one or more external plugins to provide critical
> functionality.
> >
>
> oh no !
> IMO unless there's a good reason to do so , that's a waste of time .
> That's what trac-hacks are there for .
>
> --
> Regards,
>
> Olemis.
>
> Blog ES: http://simelo-es.blogspot.com/
> Blog EN: http://simelo-en.blogspot.com/
>
> Featured article:
>
Re: Tests and CI (was: A word on Release 2)
Posted by Olemis Lang <ol...@gmail.com>.
On 10/10/12, Ryan Ollos <ry...@wandisco.com> wrote:
> On Wed, Oct 10, 2012 at 1:27 AM, Peter Koželj <pe...@digiverse.si> wrote:
>
>> This my get a bit off topic...
>>
>> I agree that testing is important, and we probably agree that testing is
>> hard.
>> In fact a good comprehensive test of a feature can take more effort to
>> write and maintain that the actual code that implements that feature.
+1
>> So
>> if
>> plugins do not provide it's own tests and we need to write and maintain
>> the
>> tests for them, the benefit of using them get's smaller.
>>
Well . The real benefit is that we'll be able to sleep at night
knowing that if some change is introduced (Bloodhound core ,
trac-hacks , plugins ...) and causes some trouble then we are just one
day away to figure it out ... especially if CI is installed .
;)
For me the benefit exists , but we certainly need to assess the
additional workload involved and write tests when necessary .
>> I still need to come to terms
>> with the idea that such a core functionality as ticket relations or multi
>> products needs to be a plugin or even worse an external plugin! With all
>> the features I would like to see in a issue tracker, I just can not
>> imagine
>> how this will work without complex inter-plugin dependencies,
>> extremely cooperational plugin authors
Notice that we are plugin maintainers of many plugins @ t-h.o , so no big deal
<joke>
... and I swear I'll cooperate with Ryan as long as at any given time
the number of test cases he writes is smaller than those I wrote ...
</joke>
>> and last but no least, all the
>> time
>> in the world :)
>>
Oh no ! No way . Exactly the same time as we were doing it using a
plugin delivered by Bloodhound itself . In case plugin maintainers are
not responsive and not willing to grant maintainership ... is not a
chaos ... we can fork the project if there's a merit
>
> I agree that you may not want the critical Bloodhound functionality to live
> externally, and to rely on developers not involved with the Bloodhound
> project to maintain it. It might be better to have any critical
> functionality be written as a plugin that lives in the Bloodhound
Indeed , we *NEED* plugin authors . There's no community without them
;)
> The idea of the functionality not being a plugin is more
> difficult for me to imagine.
>
definitely sure . Trac itself , I mean the core (tickets , vcs , ...)
maybe packaged as separate plugins as well .
> Not writing it as a plugin means you are
> just modifying existing Components in Trac (such as trac.ticket.*) to
> provide your functionality, and this would make merging from the Trac
> mainline more difficult, and in which case the tests potentially become
> even more important.
>
Interesting . Not necessarily harder . That depends . Indeed we have
overridden TicketModule but in a clever way (Gary is guilty for that
;) so as make merge process easier .
> A feature like ticket dependencies could be written as a plugin (i.e.
> Component) that is maintained within Bloodhound and is always enabled in
> the Bloodhound core, which might get around some of your concerns about
> relying on one or more external plugins to provide critical functionality.
>
oh no !
IMO unless there's a good reason to do so , that's a waste of time .
That's what trac-hacks are there for .
--
Regards,
Olemis.
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
Featured article:
Re: Tests and CI (was: A word on Release 2)
Posted by Joe Dreimann <jo...@wandisco.com>.
On 10 Oct 2012, at 14:26, Ryan Ollos <ry...@wandisco.com> wrote:
> A feature like ticket dependencies could be written as a plugin (i.e.
> Component) that is maintained within Bloodhound and is always enabled in
> the Bloodhound core, which might get around some of your concerns about
> relying on one or more external plugins to provide critical functionality.
I can't judge the technical merit of this, but conceptually it makes sense to me. Currently core plugins are probably too easy to switch off in the admin interface to be considered "always enabled".
- Joe
Re: Tests and CI (was: A word on Release 2)
Posted by Ryan Ollos <ry...@wandisco.com>.
On Wed, Oct 10, 2012 at 1:27 AM, Peter Koželj <pe...@digiverse.si> wrote:
> This my get a bit off topic...
>
> I agree that testing is important, and we probably agree that testing is
> hard.
> In fact a good comprehensive test of a feature can take more effort to
> write and maintain that the actual code that implements that feature. So if
> plugins do not provide it's own tests and we need to write and maintain the
> tests for them, the benefit of using them get's smaller.
>
> And if I dare to get a bit more of topic, I still need to come to terms
> with the idea that such a core functionality as ticket relations or multi
> products needs to be a plugin or even worse an external plugin! With all
> the features I would like to see in a issue tracker, I just can not imagine
> how this will work without complex inter-plugin dependencies,
> extremely cooperational plugin authors and last but no least, all the time
> in the world :)
>
I agree that you may not want the critical Bloodhound functionality to live
externally, and to rely on developers not involved with the Bloodhound
project to maintain it. It might be better to have any critical
functionality be written as a plugin that lives in the Bloodhound
repository. The idea of the functionality not being a plugin is more
difficult for me to imagine.
If Bloodhound is not going to permanently fork from Trac (i.e. never merge
in changes from the Trac core), then Trac is still external, and any
functionality is likely to be written as as plugin (i.e. a separate module
that inherits from Component) for Trac, which should have tests, and
specifically integration tests. Not writing it as a plugin means you are
just modifying existing Components in Trac (such as trac.ticket.*) to
provide your functionality, and this would make merging from the Trac
mainline more difficult, and in which case the tests potentially become
even more important.
A feature like ticket dependencies could be written as a plugin (i.e.
Component) that is maintained within Bloodhound and is always enabled in
the Bloodhound core, which might get around some of your concerns about
relying on one or more external plugins to provide critical functionality.
Re: Tests and CI (was: A word on Release 2)
Posted by Peter Koželj <pe...@digiverse.si>.
This my get a bit off topic...
I agree that testing is important, and we probably agree that testing is
hard.
In fact a good comprehensive test of a feature can take more effort to
write and maintain that the actual code that implements that feature. So if
plugins do not provide it's own tests and we need to write and maintain the
tests for them, the benefit of using them get's smaller.
And if I dare to get a bit more of topic, I still need to come to terms
with the idea that such a core functionality as ticket relations or multi
products needs to be a plugin or even worse an external plugin! With all
the features I would like to see in a issue tracker, I just can not imagine
how this will work without complex inter-plugin dependencies,
extremely cooperational plugin authors and last but no least, all the time
in the world :)
Peter
On Wed, Oct 10, 2012 at 7:26 AM, Ryan Ollos <ry...@wandisco.com> wrote:
> On Tue, Oct 9, 2012 at 8:16 PM, Olemis Lang <ol...@gmail.com> wrote:
>
> > [...]
> >
> > What do you mean exactly ? Something like using Gherkin (e.g. Lettuce)
> > to describe user stories ?
> >
>
> That's not what I had in mind, but I have a general idea of what you are
> talking about and also think that is a good idea to pursue at some point.
> I'm thinking, if we are going to use Trac plugins in Bloodhound, then those
> plugins need to be tested. One way to go about this is - we install and
> manually test them, decide they are good enough and bundle them into the
> application. Then, when there is a new version we want that fixes a defect,
> we code review the changes, pull in the new version and do all the manual
> testing again. Or, we write some functional tests when the plugin is
> included in Bloodhound the first time, so that we gain confidence that
> whatever we are pulling in doesn't introduce regressions. Some manual
> testing still needs to be done of course, but it would be nice to at least
> have some minimal automated functional tests ... or at least the framework
> in place to develop them at the appropriate time, which is why I raised the
> question about the framework. If I'm going to propose that we include a
> plugin in Bloodhound and I have a framework to use, I'll write some
> functional tests. You can hold me to that ;)
>
> That is for a little later though. The reason I'm raising the issue at this
> moment, is that I was fixing some defects in MasterTicketsPlugin, and I'd
> like to write some functional tests as part of that process. Probably the
> best thing to do right now, since this is outside of Bloodhound still, is
> to utilize Twill support in Trac, as you suggest.
>
>
> > [...]
> >
> > Well , it's not quite a big deal for the time being . IMO the first /
> > cheapeast step towards testing + CI is to run Trac test suite with
> > Bloodhound patches applied .
>
>
> +1, for making this the first step.
>
>
> > [...]
> >
> > If you mention that after reviewing Trac tests , yes ... Trac
> > unitest-style is a PITA ... that's why I wrote my own framework in the
> > end
> > ;)
>
>
> Well, I mention after having written some tests when submitting patches to
> the Trac core. I guess it wasn't all that bad, but I suspect there is
> framework that allows more concise and readable tests. I'm not discounting
> your way of doing it because I haven't taken a close look at what you've
> done with Twill yet, but I think it is worth getting several opinions and
> trying some different options early on, because it is setting the stage for
> a lot of work that will go into Bloodhound.
>
> Thanks for the links. I'll read-up and follow-up here. :)
>
Re: Tests and CI (was: A word on Release 2)
Posted by Ryan Ollos <ry...@wandisco.com>.
On Tue, Oct 9, 2012 at 8:16 PM, Olemis Lang <ol...@gmail.com> wrote:
> [...]
>
> What do you mean exactly ? Something like using Gherkin (e.g. Lettuce)
> to describe user stories ?
>
That's not what I had in mind, but I have a general idea of what you are
talking about and also think that is a good idea to pursue at some point.
I'm thinking, if we are going to use Trac plugins in Bloodhound, then those
plugins need to be tested. One way to go about this is - we install and
manually test them, decide they are good enough and bundle them into the
application. Then, when there is a new version we want that fixes a defect,
we code review the changes, pull in the new version and do all the manual
testing again. Or, we write some functional tests when the plugin is
included in Bloodhound the first time, so that we gain confidence that
whatever we are pulling in doesn't introduce regressions. Some manual
testing still needs to be done of course, but it would be nice to at least
have some minimal automated functional tests ... or at least the framework
in place to develop them at the appropriate time, which is why I raised the
question about the framework. If I'm going to propose that we include a
plugin in Bloodhound and I have a framework to use, I'll write some
functional tests. You can hold me to that ;)
That is for a little later though. The reason I'm raising the issue at this
moment, is that I was fixing some defects in MasterTicketsPlugin, and I'd
like to write some functional tests as part of that process. Probably the
best thing to do right now, since this is outside of Bloodhound still, is
to utilize Twill support in Trac, as you suggest.
> [...]
>
> Well , it's not quite a big deal for the time being . IMO the first /
> cheapeast step towards testing + CI is to run Trac test suite with
> Bloodhound patches applied .
+1, for making this the first step.
> [...]
>
> If you mention that after reviewing Trac tests , yes ... Trac
> unitest-style is a PITA ... that's why I wrote my own framework in the
> end
> ;)
Well, I mention after having written some tests when submitting patches to
the Trac core. I guess it wasn't all that bad, but I suspect there is
framework that allows more concise and readable tests. I'm not discounting
your way of doing it because I haven't taken a close look at what you've
done with Twill yet, but I think it is worth getting several opinions and
trying some different options early on, because it is setting the stage for
a lot of work that will go into Bloodhound.
Thanks for the links. I'll read-up and follow-up here. :)
Re: Tests and CI (was: A word on Release 2)
Posted by Olemis Lang <ol...@gmail.com>.
On 10/9/12, Ryan Ollos <ry...@wandisco.com> wrote:
> On Fri, Oct 5, 2012 at 10:52 PM, Olemis Lang <ol...@gmail.com> wrote:
>
>> [...]
>>
>> Besides IMO we should take some time to start writing tests and setup
>> CI . This is mainly considering the fact that the number of
>> contributors has increased recently and more concurrent changes
>> against a more complex implementation means a higher probability to
>> e.g. introduce regressions.
>>
>
> Additionally, it would be very nice to have some functional/acceptance
> tests for plugins that we pull from a third-party source (primarily
> trac-hacks, I imagine). The acceptance test suite might be pretty small to
> start with, but having the framework in place at least allows regressions
> to be dealt with by putting a test in place. Some recent work was leading
> me to consider this - a little off topic, but I'll circle back around here
> in a second ;)
>
What do you mean exactly ? Something like using Gherkin (e.g. Lettuce)
to describe user stories ?
If that's the case I agree ... *when* this is worthy , of course .
Otherwise , I look forward to know what are your ideas about .
[...]
> I feel like it is getting
> better, but it would sure be nice to have some functional tests at this
> stage, and even more so before pulling a plugin like this into the
> Bloodhound project.
>
Well , it's not quite a big deal for the time being . IMO the first /
cheapeast step towards testing + CI is to run Trac test suite with
Bloodhound patches applied .
> So finally, my question. The Trac project uses Twill for functional
> testing. I see some tests in bhdashboard, but they appear to be unit tests
> from what I can tell.
Let me check ... oh yes !
Now I see . That's the way I write doctests for Trac plugins . All of
them rely on a custom mini-framework built on top of dutest module
[1]_ , but the version I used is not publicly available yet .
> Has there been any discussion or exploration of what
> tool(s) to use for functional testing in Bloodhound?
nope .
> I have a very small
> amount of experience writing tests in Twill,
fwiw Trac uses twill , so you can learn if you want by reviewing their code
;)
> so I have no idea if that is a
> relatively good suite to use.
IMO , yes .
> Is there anyone with more experience that can
> provide some suggestions? I was planning to try setting up some tests in
> Twill for MasterTicketsPlugin and talk about that more when I raise that
> issue for discussion soon, but if someone has a better suggestion I would
> give it a try.
>
If you ask me go with twill ... that should be more than enough and
you'll also be able to reuse Trac shared fixture infrastructure , and
everything else they have been building since time ago .
;)
.. [1] dutest @ PyPI
(http://pypi.python.org/pypi/dutest)
--
Regards,
Olemis.
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
Featured article:
Re: Tests and CI (was: A word on Release 2)
Posted by Olemis Lang <ol...@gmail.com>.
On 10/9/12, Olemis Lang <ol...@gmail.com> wrote:
> On 10/9/12, Ryan Ollos <ry...@wandisco.com> wrote:
>> On Tue, Oct 9, 2012 at 3:43 AM, Gary Martin
>> <ga...@wandisco.com>wrote:
>>
>>> I am not sure if I am missing something but isn't selenium the usual
>>> tool
>>> for functional testing of web applications these days?
>
> There are a lot of flavors to choose from in Python . Trust me . I've
> been subscribed to TiP ML since years ago and there are many options
> at hand .
>
You can get a better idea by consulting the Python testing taxonomy [1]_ [2]_
[...]
>
>>
>> I didn't realize the Twill project wasn't active, but not suggesting we
>> use
>> it either; just noting that is what the Trac project uses. I found it to
>> be
>> a bit burdensome to write even simple functional tests.
>
> If you mention that after reviewing Trac tests , yes ... Trac
> unitest-style is a PITA ... that's why I wrote my own framework in the
> end
> ;)
>
I made this precision because writing functional tests using twill ,
wsgi_intercept and doctests is as simple as this [3]_
.. [1] PythonTestingToolsTaxonomy
(http://wiki.python.org/moin/PythonTestingToolsTaxonomy)
.. [2] PythonTestingToolsTaxonomy
(http://pycheesecake.org/wiki/PythonTestingToolsTaxonomy)
.. [3] In-process Web app testing with twill, wsgi_intercept and doctest
(http://agiletesting.blogspot.com/2006/04/in-process-web-app-testing-with-twill.html)
--
Regards,
Olemis.
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
Featured article:
Re: Tests and CI (was: A word on Release 2)
Posted by Olemis Lang <ol...@gmail.com>.
On 10/9/12, Ryan Ollos <ry...@wandisco.com> wrote:
> On Tue, Oct 9, 2012 at 3:43 AM, Gary Martin
> <ga...@wandisco.com>wrote:
>
>> I am not sure if I am missing something but isn't selenium the usual tool
>> for functional testing of web applications these days?
There are a lot of flavors to choose from in Python . Trust me . I've
been subscribed to TiP ML since years ago and there are many options
at hand .
BTW , Twill and Selenium can be used altogether [1]_ .
>> And someone has
>> just
>> pointed out to me that twill's last release was in 2007.
>>
I'll forward this question onto TiP ML to know more about the subject
. AFAICR , part of it became wsgi-intercept project and was actively
maintained the last time I took a look at it (... at the time hosted
by Google Code ... somewhere ...) , but I'm not pretty sure of its
status now .
>
> I didn't realize the Twill project wasn't active, but not suggesting we use
> it either; just noting that is what the Trac project uses. I found it to be
> a bit burdensome to write even simple functional tests.
If you mention that after reviewing Trac tests , yes ... Trac
unitest-style is a PITA ... that's why I wrote my own framework in the
end
;)
PS: BTW , twill is more than yet another testing framework ... Indeed
it becomes such a thing when wsgi-intercept is activated .
.. [1] An Introduction to Testing Web Applications with twill and Selenium
(http://books.google.com.cu/books?isbn=0596527802)
--
Regards,
Olemis.
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
Featured article:
Re: Tests and CI (was: A word on Release 2)
Posted by Ryan Ollos <ry...@wandisco.com>.
On Tue, Oct 9, 2012 at 3:43 AM, Gary Martin <ga...@wandisco.com>wrote:
> I am not sure if I am missing something but isn't selenium the usual tool
> for functional testing of web applications these days? And someone has just
> pointed out to me that twill's last release was in 2007.
>
I didn't realize the Twill project wasn't active, but not suggesting we use
it either; just noting that is what the Trac project uses. I found it to be
a bit burdensome to write even simple functional tests. I haven't worked
with Selenium, but it's obvious very popular. This looks pretty nice:
http://testutils.org/sst/. Apache licensed ;)
Re: Tests and CI (was: A word on Release 2)
Posted by Olemis Lang <ol...@gmail.com>.
On 10/9/12, Gary Martin <ga...@wandisco.com> wrote:
> On 09/10/12 10:21, Ryan Ollos wrote:
>
[...]
> And someone
> has just pointed out to me that twill's last release was in 2007.
>
I got a hint @ TiP ML . We shall take a look at retwill [1]_
.. [1] retwill: a fork from twill
(https://bitbucket.org/brandizzi/retwill)
--
Regards,
Olemis.
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
Featured article:
Re: Tests and CI (was: A word on Release 2)
Posted by Gary Martin <ga...@wandisco.com>.
On 09/10/12 10:21, Ryan Ollos wrote:
> So finally, my question. The Trac project uses Twill for functional
> testing. I see some tests in bhdashboard, but they appear to be unit
> tests from what I can tell. Has there been any discussion or
> exploration of what tool(s) to use for functional testing in
> Bloodhound? I have a very small amount of experience writing tests in
> Twill, so I have no idea if that is a relatively good suite to use. Is
> there anyone with more experience that can provide some suggestions? I
> was planning to try setting up some tests in Twill for
> MasterTicketsPlugin and talk about that more when I raise that issue
> for discussion soon, but if someone has a better suggestion I would
> give it a try.
I am not sure if I am missing something but isn't selenium the usual
tool for functional testing of web applications these days? And someone
has just pointed out to me that twill's last release was in 2007.
Cheers,
Gary