Re: Project participation and hackability

[Ross Gardler <rg...@apache.org>]

Nicola Ken Barozzi wrote:
> The goal is to increase developer participation and involvment in the 
> Forrest Project, by reducing the barrier to entry.

I like that goal, you have my attention.

> There is another point which I find very important:
> 
>                _hackability_
> 
> A project should be hackable, so that users can *quickly* tweak the code 
> and make patches, changes, enhancements.

...

>  - modular
> 
>    In this way it can be hacked in small chunks, which are more
>    easily undertstandable. Sitemap separation has done something,
>    but the locationmap will be a big improvement, separating
>    location resolving and transformation processing

In addition there is still a fair bit of stuff in core that could be 
moved out to a plugin. I've recently been thinking that core should be 
nothing more than a framework to support plugins. That is core itself is 
only responsible for ensuring the data goes to the right place at the 
right time, there is no actual functionality in there.

This is in line with the goal of making views part of core whilst 
leaving the viewHelpers as a new type of plugin.

This is exactly what the Eclipse platform is. You can download the 
platform and have nothing but an empty desktop application whose only 
capability is to download and install plugins to make it do what you want.

Anyone who has built an application on Eclipse will know just how 
powerful this is.

> Imagine that you have Forrest in production, and that you find a bug. 
> Ok, so you look in the code, find it, and fix it. Now what? You have to 
> send a patch... 

...

> It's tooooooo long and tedious. In many cases I have resisted 
> sending patches for lack of time.

...

> "What the heck" - I thought - "too much hassle. If I could just use the 
> trunk in production, it would be much easier".
> 
> Imagine that all cutting-edge users can use the trunk in production. A 
> patch is just a couple of actions away. And after incorporation, the 
> check is instantaneous, and on a *real* test, as it actually get used.

+1

========================================================================

This section is Random Thoughts
-------------------------------

The particular strength of Forrest being a framework for a collection of 
plugins is that a user need not be responsible for updating from SVN. 
With the new versioned plugin code a user can opt to stick with a stable 
version, or can opt to use the latest dev version.

If can use this to add:

- an auto update facility (say every X builds)
- an auto test after the update is done (tests using the a real site)
- an auto reporting of failed tests
- auto reverting of failed plugins

What we end up with is a user installed system that is will always be 
the latest version of everything (that works).

We can even add a facility that detects when the user has modified a 
plugin locally and (with the users permission) creates a patch and 
submits it to Jira for them. Now I reckon that makes it easier enough. 
Nicolas patch submission scenario becomes:

1. do the fix
2. "forrest site -Dforrest.updatePlugins=true"
	- the system notices a change and creates a patch file against
	  the latest svn code
3. user is asked if the patch can be submitted and provides a comment

It is likely that many of these patches will be hacks and will not be 
applied "as-is", maybe we will want to add code to turn a new feature 
on/off. But it does mean we can see what our users want the system to 
do, and they don't even have to write an email.

(ho-hum this is a small part of what I did in my PhD studies I have a 
450 page thesis on the implications of this kind of system, but I'll 
summarise in just three words - "lets do it" - if anyone is interested 
in using this technique to build software automatically take a look at 
http://www.saafe.org - this hasn't been updated for a long time, but the 
basic idea is there)

Clearly the tests do not verify that the pages are rendered correctly 
(in terms of what the human sees). For this we need a new testing 
framework. This would be brilliant to have, but I don't think it will 
arrive soon (if anyone is interested Cocoon have been looking at 
automated HTML unit tests recently, their dev lists should turn up some 
interesting info).

End of Random Thoughts
----------------------
=======================================================================

> This can only work if the trunk is always *usable*, and not only 
> _buildable_. This can make our trunk be really bug-free, as there would 
> really be a lot of eyes looking at the code.
> 
> I would thus propose that the trunk be always *releaseable* at all 
> times, and that all new functionality that interferes with usage of 
> prior features be developed on separate branches.

My random thoughts above are clearly a medium to long term goal, however 
keeping trunk bug free can be achieved immediately. The question is, how?

Do we *always* have a branch for development work, i.e. should we create 
a 0.8-dev branch now and keep trunk for a possible 0.7.1 release? The 
implications of this, that I can see, are that patches to 0.7 would also 
have to be applied to 0.8-dev, but there shouldn't be a huge volume of 
these.

> Furthremore, these branches should merge whenever possible between them 
> in a single branch so that they can be coded together, and get merged 
> with the trunk only when all developer-known bugs are fixed.

I'm not sure I follow how this will work. Can you provide an example?

> This will also make it easier for us to release often, and to learn to 
> make smaller reversible changes rather than big ones that are hard to 
> understand by other developers and users.

Yes, I like the idea of something like the following release schedule 
(this may be incomplete, I'm doing it from memory):

0.8M1  - locationmap
0.8M2  - refactored core
0.8M3  - core as a plugin framework (assuming this is agreed upon ;-)
0.8RC1 - views (which incorporate the move to XHTML2 subset)
0.8

If I understand what Nicola is saying each of these milestones would 
become trunk at their release.

I note this release schedule would mean we could release 0.8M1 in a few 
weeks time since the locationmap is very nearly there. This is a pretty 
good argument for what Nicola (and Tim?) is saying regarding developing 
in a branch then merging into trunk.

> Let me know what you think.

In principle I love it, especially the more managed release schedule. A 
couple of requests for more detail though:

How do we decide when we branch and when we work in trunk.

How do we manage dependencies between different aspects of new work. For 
example, Thorsten moved the views work to the locationmap branch because 
he needed some of the locationmap work. However, now that seems to be 
holding up the (easy) merging of the locationmap branch. What would be 
the proper way of handling this?

Ross

Re: Project participation and hackability

[Nicola Ken Barozzi <ni...@apache.org>]

David Crossley wrote:
...
> I would prefer a constantly reliable trunk. Small changes get made on
> trunk. See my other reply in this thread about test-before-commit.

Here it is:

"
 > I hope that you are still talking about the Branch-When-Needed system.
 > http://svn.collab.net/repos/svn/trunk/doc/user/svn-best-practices.html
 >
 > That is the closest to what we have been doing until now.
"

Yes, the point is: when is it _needed_? (like you ask below)
When changes are known not to be bug-free by the developer himself.

"
 > However, we are not diligent enough at the test-before-commit phase.
 > Forrestbot on the server would assist, but not replace that need.
"

Testing is good, but if changes are too big, testing is more difficult 
to do.

> Any substantial work gets done on a quick branch, e.g. our recent
> docs-reorg-branch where we worked to get it finished and merged back ASAP.
> 
> What constitutes a substantial change?

Simple. Every change that cannot be considered complete in a single 
commit, and that still has bugs that are known to the developer.

For example, Ross has said that there are still a couple of known 
problems in the locationmap branch. We cannot merge things we know are 
not ready. Same thing for the views, that are still in need of refinement.

We could say that the trunk should always be beta quality, that is 
already alpha-tested by developers.

>>>Furthremore, these branches should merge whenever possible between them 
>>>in a single branch so that they can be coded together, and get merged 
>>>with the trunk only when all developer-known bugs are fixed.
>>
>>I'm not sure I follow how this will work. Can you provide an example?

You and Thorsten know that your branches have bugs, but you need/want to 
merge. So do it, but on a locationmap-views branch. When you both don't 
have the knowledge of bugs on the previously used features, you can then 
merge with the trunk.

>>>This will also make it easier for us to release often, and to learn to 
>>>make smaller reversible changes rather than big ones that are hard to 
>>>understand by other developers and users.
>>
>>Yes, I like the idea of something like the following release schedule 
>>(this may be incomplete, I'm doing it from memory):
>>
>>0.8M1  - locationmap
>>0.8M2  - refactored core
>>0.8M3  - core as a plugin framework (assuming this is agreed upon ;-)
>>0.8RC1 - views (which incorporate the move to XHTML2 subset)
>>0.8
> 
> Why not just 0.8, 0.9, 0.10, 0.11, etc?

+1

>>If I understand what Nicola is saying each of these milestones would 
>>become trunk at their release.
> 
> I thought that trunk is the release. All that we need to do is
> freeze it for the release process.

Bingo.

releasable != can be zipped and released as-is
releasable == can be frozen and pushed to release after beta testing

In this way Forrest developers can decide to release Forrest at any 
time, without having to wait that a developer of a new feature fixes 
bugs or having to spend time understanding something that is still broken.

> What i would really like, is to not need to maintain release branches
> as we are now.

Exactly.

>>I note this release schedule would mean we could release 0.8M1 in a few 
>>weeks time since the locationmap is very nearly there. This is a pretty 
>>good argument for what Nicola (and Tim?) is saying regarding developing 
>>in a branch then merging into trunk.
>>
>>>Let me know what you think.
>>
>>In principle I love it, especially the more managed release schedule. A 
>>couple of requests for more detail though:
>>
>>How do we decide when we branch and when we work in trunk.
> 
> Yep, the big question. Perhaps trust each committer to know
> when it is appropriate. If committers are unsure, then ask.

+1

>>How do we manage dependencies between different aspects of new work. For 
>>example, Thorsten moved the views work to the locationmap branch because 
>>he needed some of the locationmap work. However, now that seems to be 
>>holding up the (easy) merging of the locationmap branch. What would be 
>>the proper way of handling this?
> 
> Not to get into that situation again. The Locationmap branch was away
> too long and the release process was delayed too much.

It can happen when features are so important and "core" to Forrest. In 
general it would not happen.


-- 
Nicola Ken Barozzi                   nicolaken@apache.org
             - verba volant, scripta manent -
    (discussions get forgotten, just code remains)
---------------------------------------------------------------------

Re: Project participation and hackability

[Ross Gardler <rg...@apache.org>]

David Crossley wrote:
> Ross Gardler wrote:
> 
>>Nicola Ken Barozzi wrote:
>>

...

>>>This will also make it easier for us to release often, and to learn to 
>>>make smaller reversible changes rather than big ones that are hard to 
>>>understand by other developers and users.
>>
>>Yes, I like the idea of something like the following release schedule 
>>(this may be incomplete, I'm doing it from memory):
>>
>>0.8M1  - locationmap
>>0.8M2  - refactored core
>>0.8M3  - core as a plugin framework (assuming this is agreed upon ;-)
>>0.8RC1 - views (which incorporate the move to XHTML2 subset)
>>0.8
> 
> 
> Why not just 0.8, 0.9, 0.10, 0.11, etc?

No reason, my mind is just stuck in the current roadmap we have on Jira.

+1 for 0.8, 0.9.

Ross

Re: Project participation and hackability

[David Crossley <cr...@apache.org>]

Ross Gardler wrote:
> Nicola Ken Barozzi wrote:
> 
> >This can only work if the trunk is always *usable*, and not only 
> >_buildable_. This can make our trunk be really bug-free, as there would 
> >really be a lot of eyes looking at the code.
> >
> >I would thus propose that the trunk be always *releaseable* at all 
> >times, and that all new functionality that interferes with usage of 
> >prior features be developed on separate branches.
> 
> My random thoughts above are clearly a medium to long term goal, however 
> keeping trunk bug free can be achieved immediately. The question is, how?
> 
> Do we *always* have a branch for development work, i.e. should we create 
> a 0.8-dev branch now and keep trunk for a possible 0.7.1 release? The 
> implications of this, that I can see, are that patches to 0.7 would also 
> have to be applied to 0.8-dev, but there shouldn't be a huge volume of 
> these.

Sounds like too many branches already. I would prefer a constantly
reliable trunk. Small changes get made on trunk. See my other reply
in this thread about test-before-commit.

Any substantial work gets done on a quick branch, e.g. our recent
docs-reorg-branch where we worked to get it finished and merged back ASAP.

What constitutes a substantial change?

> >Furthremore, these branches should merge whenever possible between them 
> >in a single branch so that they can be coded together, and get merged 
> >with the trunk only when all developer-known bugs are fixed.
> 
> I'm not sure I follow how this will work. Can you provide an example?
> 
> >This will also make it easier for us to release often, and to learn to 
> >make smaller reversible changes rather than big ones that are hard to 
> >understand by other developers and users.
> 
> Yes, I like the idea of something like the following release schedule 
> (this may be incomplete, I'm doing it from memory):
> 
> 0.8M1  - locationmap
> 0.8M2  - refactored core
> 0.8M3  - core as a plugin framework (assuming this is agreed upon ;-)
> 0.8RC1 - views (which incorporate the move to XHTML2 subset)
> 0.8

Why not just 0.8, 0.9, 0.10, 0.11, etc?

> If I understand what Nicola is saying each of these milestones would 
> become trunk at their release.

I thought that trunk is the release. All that we need to do is
freeze it for the release process.

What i would really like, is to not need to maintain release branches
as we are now.

> I note this release schedule would mean we could release 0.8M1 in a few 
> weeks time since the locationmap is very nearly there. This is a pretty 
> good argument for what Nicola (and Tim?) is saying regarding developing 
> in a branch then merging into trunk.
> 
> >Let me know what you think.
> 
> In principle I love it, especially the more managed release schedule. A 
> couple of requests for more detail though:
> 
> How do we decide when we branch and when we work in trunk.

Yep, the big question. Perhaps trust each committer to know
when it is appropriate. If committers are unsure, then ask.

> How do we manage dependencies between different aspects of new work. For 
> example, Thorsten moved the views work to the locationmap branch because 
> he needed some of the locationmap work. However, now that seems to be 
> holding up the (easy) merging of the locationmap branch. What would be 
> the proper way of handling this?

Not to get into that situation again. The Locationmap branch was away
too long and the release process was delayed too much.

-David