J-C oversight (was: Benefits of Apache Commons)

[Greg Stein <gs...@lyra.org>]

On Tue, Nov 11, 2003 at 03:41:25PM -0800, Rodney Waldhoff wrote:
> On Mon, 10 Nov 2003, Greg Stein wrote:
> > If you want to fix the oversight in J-C, then do so. Personally, I don't
> > believe it is possible, for the reasons that Robert has already described.
> > But I certainly won't stop you from trying.
> 
> Greg, can you expand upon what you see as the "oversight" problem in
> Jakarta-Commons?  I'm not sure what you mean by that.

This feels like it is getting off-topic, although there is still some
relevance since A-C is quite similar to J-C.

Take things like Hivemind, Jelly, and whatstheotherone. In the list that
Stephen posted recently, he labeled these as "should be TLPs." What the
*heck* are they doing way down in J-C if they are TLP material? How could
Hivemind grow to become a framework while sitting in Commons without
anybody really saying, "wow. that needs to move."

There is a "sandbox" which is labeled as some kind of playground for ASF
committers to monkey in. That is wrong. *Anything* in the ASF CVS
repository is owned by the ASF and requires the *SAME* oversight as larger
projects like Tomcat, httpd, or Cocoon. Label the sandbox all you want as
"unofficial" or "being worked on until it 'graduates'" or whatever. The
simple fact is that the mechanism encourages single committers to go wild
under the banner of the ASF.

The J-C development list is apparently so trafficked that individuals
cannot really keep up with it. To retain proper oversight, that must be
broken down and partitioned into manageable chunks. Quite doable. But the
PMC is still responsible, as a whole, for every chunk that is produced. No
matter how you might partition the *mailing lists*, that total amount of
traffic is still present. Then throw in all the other Jakarta traffic.
Then try to say that a group of a couple dozen people are directing *ALL*
of that effort. It just isn't believable.

The problem is that it has to be beyond believable. We have to be able to
stand up in a court and say "the PMC told those committers to do that."
There can't be a question about it.

Putting "one member from each (sub-)sub-project onto the PMC" is getting
there, but that still feels like the individual, rather than the PMC, is
managing the project.

The information overload is very well characterized by a reference
somebody made recently to a Jakarta report to the board. See Attachment D
in the March minutes:

    http://www.apache.org/foundation/records/minutes/2003/board_minutes_2003_03_19.txt

The most recent report defers much of the report to the Jakarta
newsletter. It doesn't summarize the state for the Board, and it doesn't
provide any insight into community issues, interactions, legal issues, or
forward thoughts of the PMC. We have this *massive* amount of code and a
*huge* community, and the PMC is not able to effectively report on what is
happening.

Is it just me? Maybe. But as a Director, I'm supposed to be able to review
this stuff and know whether Jakarta is going well/poorly. I have never
felt that I have enough information to really know that. And the scary
part is that I'm one of the more informed Directors -- I'm subscribed to
the Jakarta PMC list (along with one or two other Directors). The rest?
Scrappy info.

All that said, I can also state that it could be fairly said some of the
other PMCs might not be providing enough information either. In fact, that
has been raised here, "well, yah, but are you getting reports from the
httpd docs group?" Otherwise stated as, "but they suck, so we can too!" :-)
My point is that I may be unfairly focused on Jakarta. But I get to do
that; I'm just me, not the Board, and I get to have an opinion on where I
think the sore spot is. (and have no fear, I've got criticisms for most of
the PMCs :-)  Can any of the PMCs operate Right And True(tm) ? Dunno, but
I think that the Jakarta PMC has the lowest hope because of the sheer size
of the projects under it. And J-C is one of those, and is arguably one the
*most* active projects within Jakarta, yet it almost has an extra "layer"
between it and the PMC -- it effectively has its own charter and rules and
other policies. There have been threats of a J-C "sub PMC" which have
(thankfully) been shot down.


I'll leave the comparison/contrast against A-C to others :-)

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/

Re: J-C oversight (was: Benefits of Apache Commons)

[robert burrell donkin <rd...@apache.org>]

On 13 Nov 2003, at 01:05, Stephen Colebourne wrote:

> From: "robert burrell donkin" <rd...@apache.org>
>> jakarta-commons is high traffic but the worries about scalability are
>> not to do with supervision but whether normal developers can continue
>> to be attracted.
>
> I've been quiet for a while, but I wanted to agree with this last 
> statement.
> Its not about supervision in j-c, but keeping attracting new ideas and
> committers.
>
> I would suggest
> a) the j-c mailing list is very high traffic
> b) some newcomers/contributers are definitely being put off by this
> c) the ability for j-c to accept new components must eventually be 
> limited
> by the one mailing list arrangement
> d) j-c is very well supervised
> e) committed coders are in charge of their components
>
> My mail system would undoutably be better off with multiple lists. But 
> j-c
> may suffer. This is a dilema. It would be nice if there was a clear 
> division
> into two or three that could occur, but finding it causes problems.

i'd be very, very unhappy to see any more sub-projects with split 
mailing lists at jakarta. jakarta is moving towards a more healthy 
position but it's going to take time. split mailing lists are part of 
the problem. we've got enough problems at jakarta without making any 
more.

so, i'd like to try to encourage people to think about whether there's 
enough synergy to move some products (at least) here to apache commons. 
the folks already here are all library builders (even though in other 
languages) so there's at least one area of common interest. some 
jakarta-commons products share similar goals. the ASF also contains 
some of the original framers of RFCs that some commons products are 
implementing.

in terms of synergy in the community, i'd say that it's early enough in 
the process (of formation) for the apache commons community to be 
influenced by any product communities who want to move. there's 
unlikely to be too much repressive beurocracy here (almost certainly 
less than the jakarta pmc is likely to be forced to institute - jakarta 
is now too big and too diverse.)

it seems to me that people who haven't been around jakarta for too long 
tend to forget that when jakarta was formed, jakarta was very 
frequently, strongly (and unfairly) critisized for living off the 
reputation of the apache httpd server. in the end, the apache way (and 
the quality of the jakarta code and the jakarta people) prevailed. it 
might seem strange now but it's very possible that apache commons might 
become (in a few years time) as reknown as a producers of library code 
as jakarta is reknown as a producer of server-side code. but only if 
those people who help to make jakarta-commons such a hothouse decide to 
move here.

IMHO the jakarta pmc is going to have to get a lot tougher and it's 
going to become a little more painful for committers than it's been 
before. we're going to have to take even more seriously the issues of 
scope and oversight. this means that some stuff that (in other 
circumstances) would be really cool might not be able to happen at 
jakarta. this doesn't mean that it shouldn't happen elsewhere in 
apache.

- robert

Divinding J-C

[Stephen Colebourne <sc...@btopenworld.com>]

> My mail system would undoutably be better off with multiple lists. But j-c
> may suffer. This is a dilema. It would be nice if there was a clear
division
> into two or three that could occur, but finding it causes problems.
>
There MAY be potential to divide j-c into two separate Jakarta projects,
'core' and 'utility'. The division I see based on level in an (example)
application stack:

User application
Struts etc.
Utility
Core

Thus utility depends on core jar files, but not vice versa. In fact, core
jars generally depend on nothing but the JDK. The utility level components
also tend to have more 'religion' about them (ie. you have to take more time
to learn them, and to fit into their ways).

This kind of division would give j-c more sensible traffic on the lists,
hence more sensible and clear oversight. Because the division is only into
two, there should be enough people about to populate each sufficiently.

Well, its an idea (and probably should go to j-c list ;-)
Stephen

----- Original Message -----
From: "Stephen Colebourne" <sc...@btopenworld.com>
To: <ge...@commons.apache.org>
Sent: Thursday, November 13, 2003 1:05 AM
Subject: Re: J-C oversight (was: Benefits of Apache Commons)


> From: "robert burrell donkin" <rd...@apache.org>
> > jakarta-commons is high traffic but the worries about scalability are
> > not to do with supervision but whether normal developers can continue
> > to be attracted.
>
> I've been quiet for a while, but I wanted to agree with this last
statement.
> Its not about supervision in j-c, but keeping attracting new ideas and
> committers.
>
> I would suggest
> a) the j-c mailing list is very high traffic
> b) some newcomers/contributers are definitely being put off by this
> c) the ability for j-c to accept new components must eventually be limited
> by the one mailing list arrangement
> d) j-c is very well supervised
> e) committed coders are in charge of their components
>
> Stephen
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@commons.apache.org
> For additional commands, e-mail: general-help@commons.apache.org
>

Re: J-C oversight (was: Benefits of Apache Commons)

[Stephen Colebourne <sc...@btopenworld.com>]

From: "robert burrell donkin" <rd...@apache.org>
> jakarta-commons is high traffic but the worries about scalability are
> not to do with supervision but whether normal developers can continue
> to be attracted.

I've been quiet for a while, but I wanted to agree with this last statement.
Its not about supervision in j-c, but keeping attracting new ideas and
committers.

I would suggest
a) the j-c mailing list is very high traffic
b) some newcomers/contributers are definitely being put off by this
c) the ability for j-c to accept new components must eventually be limited
by the one mailing list arrangement
d) j-c is very well supervised
e) committed coders are in charge of their components

My mail system would undoutably be better off with multiple lists. But j-c
may suffer. This is a dilema. It would be nice if there was a clear division
into two or three that could occur, but finding it causes problems.

Stephen

Re: J-C oversight

[Sam Ruby <ru...@apache.org>]

robert burrell donkin wrote:
> On 12 Nov 2003, at 11:21, Greg Stein wrote:
> 
> <snip>
> 
>> The J-C development list is apparently so trafficked that individuals
>> cannot really keep up with it. To retain proper oversight, that must be
>> broken down and partitioned into manageable chunks. Quite doable. But the
>> PMC is still responsible, as a whole, for every chunk that is 
>> produced. No
>> matter how you might partition the *mailing lists*, that total amount of
>> traffic is still present. Then throw in all the other Jakarta traffic.
>> Then try to say that a group of a couple dozen people are directing *ALL*
>> of that effort. It just isn't believable.
> 
> 
> i'm concerned about this kind of criticism (which has very often been 
> repeated). it doesn't really tally with the opinions expressed by the 
> jakarta pmc. jakarta-commons has more ASF members and jakarta pmc 
> members subscribed than any other jakarta sub-project. many of these 
> people look at every message that is sent to the list.
> 
> one mailing list is the solution not the problem. it forces everyone 
> close together and makes it impossible to keep anything a secret for 
> very long. in the past, the problems have arisen with umbrella 
> sub-projects. these develop their own communities and their own rules of 
> behaviour. they spawn new mailing lists and sub-sub-projects dividing 
> the community and preventing effective oversight.
> 
> what worries myself is not the oversight of the jakarta-commons. it's 
> turbine. i don't believe that umbrella sub-projects can be effectively 
> supervised.
> 
> please greg, listen to what we've been saying over the last few months 
> about this issue. the consensus on the jakarta pmc is that 
> jakarta-commons works very well and is well supervised. we are worried 
> about supervision - very worried - but not about worried at all about 
> jakarta-commons.
> 
> jakarta-commons is high traffic but the worries about scalability are 
> not to do with supervision but whether normal developers can continue to 
> be attracted.
> 
> - robert

I concur with this assessment.

- Sam Ruby

Re: J-C oversight (was: Benefits of Apache Commons)

[robert burrell donkin <rd...@apache.org>]

On 12 Nov 2003, at 11:21, Greg Stein wrote:

<snip>

> The J-C development list is apparently so trafficked that individuals
> cannot really keep up with it. To retain proper oversight, that must be
> broken down and partitioned into manageable chunks. Quite doable. But 
> the
> PMC is still responsible, as a whole, for every chunk that is 
> produced. No
> matter how you might partition the *mailing lists*, that total amount 
> of
> traffic is still present. Then throw in all the other Jakarta traffic.
> Then try to say that a group of a couple dozen people are directing 
> *ALL*
> of that effort. It just isn't believable.

i'm concerned about this kind of criticism (which has very often been 
repeated). it doesn't really tally with the opinions expressed by the 
jakarta pmc. jakarta-commons has more ASF members and jakarta pmc 
members subscribed than any other jakarta sub-project. many of these 
people look at every message that is sent to the list.

one mailing list is the solution not the problem. it forces everyone 
close together and makes it impossible to keep anything a secret for 
very long. in the past, the problems have arisen with umbrella 
sub-projects. these develop their own communities and their own rules 
of behaviour. they spawn new mailing lists and sub-sub-projects 
dividing the community and preventing effective oversight.

what worries myself is not the oversight of the jakarta-commons. it's 
turbine. i don't believe that umbrella sub-projects can be effectively 
supervised.

please greg, listen to what we've been saying over the last few months 
about this issue. the consensus on the jakarta pmc is that 
jakarta-commons works very well and is well supervised. we are worried 
about supervision - very worried - but not about worried at all about 
jakarta-commons.

jakarta-commons is high traffic but the worries about scalability are 
not to do with supervision but whether normal developers can continue 
to be attracted.

- robert

Re: J-C oversight (was: Benefits of Apache Commons)

[Rodney Waldhoff <rw...@apache.org>]

On Wed, 12 Nov 2003, Greg Stein wrote:

> On Tue, Nov 11, 2003 at 03:41:25PM -0800, Rodney Waldhoff wrote:
> > On Mon, 10 Nov 2003, Greg Stein wrote:
> > > If you want to fix the oversight in J-C, then do so. Personally, I don't
> > > believe it is possible, for the reasons that Robert has already described.
> > > But I certainly won't stop you from trying.
> >
> > Greg, can you expand upon what you see as the "oversight" problem in
> > Jakarta-Commons?  I'm not sure what you mean by that.
>
> This feels like it is getting off-topic, although there is still some
> relevance since A-C is quite similar to J-C.

Thanks for taking the time to respond here.  I disagree with some
(although not all) of these assertions, but since as you note this is
beginning to push to bounds of on-topic discussion, I won't try to debate
them here.

I do have one question regarding this:

> Take things like Hivemind, Jelly, and whatstheotherone. In the list that
> Stephen posted recently, he labeled these as "should be TLPs." What the
> *heck* are they doing way down in J-C if they are TLP material? How could
> Hivemind grow to become a framework while sitting in Commons without
> anybody really saying, "wow. that needs to move."

Just to make sure I undestand you, is your concern here that these
projects are (a) out of scope for jakarta (b) out of scope of jakarta
commons or (c) too large for jakarta commons, or (d) some combination of
the above?

>
> Cheers,
> -g
>

 - Rod <http://radio.weblogs.com/0122027/>

Re: J-C oversight

[Sam Ruby <ru...@apache.org>]

Greg Stein wrote:
> 
> The information overload is very well characterized by a reference
> somebody made recently to a Jakarta report to the board. See Attachment D
> in the March minutes:
> 
>     http://www.apache.org/foundation/records/minutes/2003/board_minutes_2003_03_19.txt
> 
> The most recent report defers much of the report to the Jakarta
> newsletter. It doesn't summarize the state for the Board, and it doesn't
> provide any insight into community issues, interactions, legal issues, or
> forward thoughts of the PMC. We have this *massive* amount of code and a
> *huge* community, and the PMC is not able to effectively report on what is
> happening.
> 
> Is it just me? Maybe. But as a Director, I'm supposed to be able to review
> this stuff and know whether Jakarta is going well/poorly. I have never
> felt that I have enough information to really know that. And the scary
> part is that I'm one of the more informed Directors -- I'm subscribed to
> the Jakarta PMC list (along with one or two other Directors). The rest?
> Scrappy info.

If I recall correctly, this report was approved as submitted.  You 
certainly could have raised issues at the time, but instead it was 
accepted by unamimous consent... only to be brought up later as an issue 
on the general@commons mailing list.

I personally think that having the information online, continuously 
updated, and available for review to all is a good thing.

One way or another, the root issue is information overload.  One could 
certainly immagine a four paragraph update for each row in the following:

http://gump.covalent.net/log/xref.html

- Sam Ruby

Re: J-C oversight (was: Benefits of Apache Commons)

[Henri Yandell <ba...@generationjava.com>]

On Wed, 12 Nov 2003, Greg Stein wrote:

> Take things like Hivemind, Jelly, and whatstheotherone. In the list that
> Stephen posted recently, he labeled these as "should be TLPs." What the
> *heck* are they doing way down in J-C if they are TLP material? How could
> Hivemind grow to become a framework while sitting in Commons without
> anybody really saying, "wow. that needs to move."

Hivemind and Math are still in the sandbox, so now is the correct time for
people to be saying 'wow. that needs to move.'. Jelly is in Commons
'proper', but has not yet released so now is also the right time for that.

Hivemind are putting together a proposal to be an SLP [what is the term
for this, S for Second] within Jakarta.

> There is a "sandbox" which is labeled as some kind of playground for ASF
> committers to monkey in. That is wrong.

It is a part of the Jakarta Commons Charter that it is an incubation area.

> *Anything* in the ASF CVS
> repository is owned by the ASF and requires the *SAME* oversight as larger
> projects like Tomcat, httpd, or Cocoon. Label the sandbox all you want as

No argument here. Being watchful over legalities in the sandbox is an
essential role of the J-Commons community.

> "unofficial" or "being worked on until it 'graduates'" or whatever. The
> simple fact is that the mechanism encourages single committers to go wild
> under the banner of the ASF.

It is less managed than Incubation and I agree that as Incubator matures
the J-Commons-Sandbox needs to get more restrictive to be more negative
towards larger concepts. I'm not sure Jelly or Math would have been
developed elsewhere under such a concept, but maybe Hivemind would have
been developed under the incubator rather than in sandbox. Effectively as
Incubator matures, the sandbox will become more and more a back door.

> The J-C development list is apparently so trafficked that individuals
> cannot really keep up with it.

I keep up happily with the parts I consider myself involved in. I've no
interest in Jelly/Hivemind/some of the others and so filter them out then
delete later.

> To retain proper oversight, that must be
> broken down and partitioned into manageable chunks. Quite doable. But the

Agreed. Even knowing that every component on the J-C list has a PMC member
is not enough to ensure that it has oversight. What if there is only one
member paying attention to one component and that person is on vacation,
or focusing attention elsewhere. Even if there were 4 out of 5 PMC'ers on
the component, how do you ensure those 4 are paying attention to what the
other 1 is doing.

> PMC is still responsible, as a whole, for every chunk that is produced. No
> matter how you might partition the *mailing lists*, that total amount of
> traffic is still present. Then throw in all the other Jakarta traffic.
> Then try to say that a group of a couple dozen people are directing *ALL*
> of that effort. It just isn't believable.
>
> The problem is that it has to be beyond believable. We have to be able to
> stand up in a court and say "the PMC told those committers to do that."
> There can't be a question about it.

"Someone on the PMC was aware of it and had the opportunity to veto it" is
where I would currently say we are. Though the veto is after the action.

> Putting "one member from each (sub-)sub-project onto the PMC" is getting
> there, but that still feels like the individual, rather than the PMC, is
> managing the project.

It seems like a typical people problem. There are too many people for the
shallow management structure that ASF like, there is also a small amount
of energy for management in a structure like ASF. The only solution so far
has been to remove people by creating more TLPs.

Another solution would be to instill more bureacracy and internal reports
inside Jakarta that filter upwards to be summarised to board. Effectively
turning the newsletter into a charter-mandated report tree.

> Is it just me? Maybe. But as a Director, I'm supposed to be able to review

Only in that you're the one who has to have the complete oversight or
trust in oversight.

> All that said, I can also state that it could be fairly said some of the
> other PMCs might not be providing enough information either. In fact, that
> has been raised here, "well, yah, but are you getting reports from the
> httpd docs group?" Otherwise stated as, "but they suck, so we can too!" :-)
> My point is that I may be unfairly focused on Jakarta. But I get to do

I agree that it needs the focus as the biggest blind-side to you. The
biggest part of the 'reports from httpd docs group' question is really 'so
show us how it's meant to be'.

> of the projects under it. And J-C is one of those, and is arguably one the
> *most* active projects within Jakarta, yet it almost has an extra "layer"
> between it and the PMC -- it effectively has its own charter and rules and
> other policies. There have been threats of a J-C "sub PMC" which have
> (thankfully) been shot down.

The bullets missed ;) PMC members who work in J-Commons are going to
become a de-facto sub-committee I suspect. In the same way that 2
committers on Jakarta Turbine [random name] are effectively the
sub-committee there.

Hen

Re: J-C oversight (was: Benefits of Apache Commons)

["Geir Magnusson Jr." <ge...@optonline.net>]

On Wednesday, November 12, 2003, at 06:21 AM, Greg Stein wrote:
>
>
> All that said, I can also state that it could be fairly said some of 
> the
> other PMCs might not be providing enough information either. In fact, 
> that
> has been raised here, "well, yah, but are you getting reports from the
> httpd docs group?" Otherwise stated as, "but they suck, so we can 
> too!" :-)

I think you are misrepresenting me again.  I'll assume it isn't 
deliberate. :)

My point was only in relation to your statement that the board didn't 
receive a j-c report from the Jakarta PMC.  To that end, I asked if 
other Apache projects gave individual reports, and used 'docs' as an 
example.  I had no belief that the board got such a report, nor that 
one was really required.  I'm assuming httpd docs are just fine.

And never will I claim that we can suck because someone else does - 
both have to be cleaned up if both suck.

geir

-- 
Geir Magnusson Jr                                   203-247-1713(m)
geirm@optonline.net