Next steps after 2.2.1 release

[Greg Trasuk <tr...@trasuk.com>]

OK, so in my last message I talked about how (speaking only for myself) I'm a little nervous about the state of the trunk.

So what now?  

Problems we need to avoid in this discussion:
-------------------------------------------------------------

- Conflation of source tree structure issues with build tool selection.
- Conflation of Maven build, Maven as codebase provider (artifact urls), and posting artifacts to Maven Central
- Wish lists of pet features
- Bruised egos and personal criticisms.

Issues I see, in no particular order:
----------------------------------------------
- We've done changes both to the test framework and the code, and lots of them.  We should do one or the other, or small amounts of coevolution, if absolutely necessary.
- Really, I'd like to see a completely separate integration test, and have the TCK tests separated out again.
- The source tree is incomprehensible
- The tests appear to be awfully sensitive to their environment.  Insofar as when I run them locally on an untouched source tree, I get 280 failures.
- There have been changes to class loading and security subsystems.  These subsystems are core to Jini, and the changes were made to the existing source, so there's no way to "opt-out" of the changes.  I'd like to see radical changes be optional until proven in the field, where possible.  In the case of policy providers and class loaders, that should be easy to do.
- Similarly, it seems there have been some changes to the JERI framework.
- There are ".jar" files in our repository.  I'll stipulate that the licensing has been checked, but it smells bad.

Discussion
-----------------
I guess the biggest thing I'd like to see is stability in the test framework.  Perhaps it needs refactoring or reorganization, but if so, we need to be very careful to separate it from changes to the core functionality.

Next, I'd like for it to be easier to comprehend the source tree.  I think a good way to do that is to separate out (carefully) the core Jini package (basically the contents of jsk-platform.jar) and the service implementations.  There's no reason that we have to have one huge everything-but-the-kitchen-sink distribution.  That's just a holdover from how Sun structured the JTSK - It was literally a "starter kit".  To me it would be fine to have separate deliverables for the platform and the services.

While we're separating out the services, it might also be a decent time to implement Maven-based builds if we think that's a good idea.  I'd start with Reggie.  It would also be a good time to get rid of the "com.sun.jini" packages.

Aside:  I'm personally ambivalent on Maven (which is to say I'm nowhere near as negative on it as I once was).  I do agree with Dennis, though, that the jars and appropriate poms need to be published to Maven Central.  There's no doubt that users will appreciate that.

Once we have a stable set of regression tests, then OK, we could think about improving performance or using Maven repositories as the codebase server.

I realize this won't be popular, but my gut feel is that we need to step back to the 2.2 branch and retrace our steps a little, and go through the evolution again in a more measured fashion.

Proposal
------------

1 - Release version 2.2.1 from the 2.2 branch.
2 - Create a separate source tree for the test framework.  This could come from the "qa_refactor" branch, but the goal should be to successfully test the 2.2.1 release.  Plus it should be a no-brainer to pull it down and run it on a local machine.
3 - Release 2.2.2 from the pruned jtsk tree.  Release 1.0.0 of the test framework.
4 - Pull out the infrastructure service implementations (Reggie, Outrigger, Norm, etc) from the core into separate products.  Release 1.0.0 on each of them.  Release 2.2.3 from the pruned jtsk tree.
5 - Adopt a fixed release cycle.  Not sure if it should be quarterly or biennial, or whether it should be all products at once or staggered releases.  We'll need to discuss.
6 - Then we can start making changes if necessary to the individual products.  And also try to deal with making it easier for new users to use the technology.

So there you go.  Opinions?

Greg Trasuk.

Re: Next steps after 2.2.1 release

[Dan Creswell <da...@gmail.com>]

Peter,

I shall remind you of your statement elsewhere about behaviour in public.
Dude, I know you're a much better person that the below suggests.

Perhaps you wrote it in anger or frustration or fatigue or some
combination. Nevertheless it doesn't come off well and would point at you
needing to do just as much development of leadership skills as you assert
is required for Greg.

Trust has to be earned just as much as granted. It starts from respect and
quality dialogue.

On 7 April 2013 22:54, Peter <ji...@zeus.net.au> wrote:

> Greg, why have you repeated this message?
>
> I think this is a deliberate attack on the project because you haven't
> been following development in trunk and now you're scared because you see
> changes you don't understand.
>
> I've been following your developments in surrogates, an impressive amount
> of productivity.  Although I think you should consider upgrading
> apache.commons vfs to version 2 before releasing.
>
> Open your mind and ask questions, the code isn't set in stone, you have an
> obligation as project lead to encourage and nurture development, not stifle
> it.
>
> You strike me as someone who's a very good programmer, but still learning
> leadership because you lack faith in others and must do everything
> yourself.  Remember I offered to assist with Surrogates, but you wanted to
> work alone?
>
> You need to let go and give others a go too.
>
> How you handle this matter will be a test for your own personal
> development and an opportunity to grow as a leader.
>
> You also hold the future of this project in your hands, so I hope you find
> strength to let go.
>
> Regards,
>
> Peter.
>
> ----- Original message -----
> >
> > OK, so in my last message I talked about how (speaking only for myself)
> I'm a
> > little nervous about the state of the trunk.
> >
> > So what now?
> >
> > Problems we need to avoid in this discussion:
> > -------------------------------------------------------------
> >
> > - Conflation of source tree structure issues with build tool selection.
> > - Conflation of Maven build, Maven as codebase provider (artifact urls),
> and
> > posting artifacts to Maven Central - Wish lists of pet features
> > - Bruised egos and personal criticisms.
> >
> > Issues I see, in no particular order:
> > ----------------------------------------------
> > - We've done changes both to the test framework and the code, and lots
> of them.
> > We should do one or the other, or small amounts of coevolution, if
> absolutely
> > necessary. - Really, I'd like to see a completely separate integration
> test, and
> > have the TCK tests separated out again. - The source tree is
> incomprehensible -
> > The tests appear to be awfully sensitive to their environment.  Insofar
> as when
> > I run them locally on an untouched source tree, I get 280 failures. -
> There have
> > been changes to class loading and security subsystems.  These subsystems
> are
> > core to Jini, and the changes were made to the existing source, so
> there's no
> > way to "opt-out" of the changes.  I'd like to see radical changes be
> optional
> > until proven in the field, where possible.  In the case of policy
> providers and
> > class loaders, that should be easy to do. - Similarly, it seems there
> have been
> > some changes to the JERI framework. - There are ".jar" files in our
> repository.
> > I'll stipulate that the licensing has been checked, but it smells bad.
> >
> > Discussion
> > -----------------
> > I guess the biggest thing I'd like to see is stability in the test
> framework.
> > Perhaps it needs refactoring or reorganization, but if so, we need to be
> very
> > careful to separate it from changes to the core functionality.
> >
> > Next, I'd like for it to be easier to comprehend the source tree.  I
> think a
> > good way to do that is to separate out (carefully) the core Jini package
> > (basically the contents of jsk-platform.jar) and the service
> implementations.
> > There's no reason that we have to have one huge
> everything-but-the-kitchen-sink
> > distribution.  That's just a holdover from how Sun structured the JTSK -
> It was
> > literally a "starter kit".  To me it would be fine to have separate
> deliverables
> > for the platform and the services.
> >
> > While we're separating out the services, it might also be a decent time
> to
> > implement Maven-based builds if we think that's a good idea.  I'd start
> with
> > Reggie.  It would also be a good time to get rid of the "com.sun.jini"
> packages.
> >
> > Aside:  I'm personally ambivalent on Maven (which is to say I'm nowhere
> near as
> > negative on it as I once was).  I do agree with Dennis, though, that the
> jars
> > and appropriate poms need to be published to Maven Central.  There's no
> doubt
> > that users will appreciate that.
> >
> > Once we have a stable set of regression tests, then OK, we could think
> about
> > improving performance or using Maven repositories as the codebase server.
> >
> > I realize this won't be popular, but my gut feel is that we need to step
> back to
> > the 2.2 branch and retrace our steps a little, and go through the
> evolution
> > again in a more measured fashion.
> >
> > Proposal
> > ------------
> >
> > 1 - Release version 2.2.1 from the 2.2 branch.
> > 2 - Create a separate source tree for the test framework.  This could
> come from
> > the "qa_refactor" branch, but the goal should be to successfully test
> the 2.2.1
> > release.  Plus it should be a no-brainer to pull it down and run it on a
> local
> > machine. 3 - Release 2.2.2 from the pruned jtsk tree.  Release 1.0.0 of
> the test
> > framework. 4 - Pull out the infrastructure service implementations
> (Reggie,
> > Outrigger, Norm, etc) from the core into separate products.  Release
> 1.0.0 on
> > each of them.  Release 2.2.3 from the pruned jtsk tree. 5 - Adopt a fixed
> > release cycle.  Not sure if it should be quarterly or biennial, or
> whether it
> > should be all products at once or staggered releases.  We'll need to
> discuss. 6
> > - Then we can start making changes if necessary to the individual
> products.  And
> > also try to deal with making it easier for new users to use the
> technology.
> >
> > So there you go.  Opinions?
> >
> > Greg Trasuk.
> >
>
>

Re: Next steps after 2.2.1 release

[Patricia Shanahan <pa...@acm.org>]

On 4/7/2013 5:03 PM, Greg Trasuk wrote:
...
> I'm honestly and truly not passing judgement on the quality of the
> code.  I honestly don't know if it's good or bad.  I have to confess
> that, given that Jini was written as a top-level project at Sun,
> sponsored by Bill Joy, when Sun was at the top of its game, and the Jini
> project team was a "who's-who" of distributed computing pioneers, the
> idea that it's riddled with concurrency bugs surprises me.  But mainly,
> I'm still trying to answer that question - "How do I know if it's good?"
...

I don't know whether it has concurrency bugs, and that is a problem in 
its own right. The theory of why does not suffer from concurrency 
problems is nowhere near clear.

I have no faith in the infallibility of Sun developers, because I used 
to be one. Some of them were very, very smart, but those were not 
necessarily the ones writing every line of code. The issue is not the 
distributed system design, but details of coding that may be leading to 
local concurrency problems within a program.

I am a little worried that my doubts about RetryTask may lead to over 
focus on that issue. It should be considered as a candidate, but I was 
never able to become certain there was a bug involving it. If I had, I 
would have created an issue for it and fixed it.

Patricia

Re: Next steps after 2.2.1 release

[Peter <ji...@zeus.net.au>]

----- Original message -----
>
> On Sun, 2013-04-07 at 17:54, Peter wrote:
> > Greg, why have you repeated this message?
> >
>
> First time I sent it was from the wrong email address, so it got hung up
> in moderation.  I sent it again from my subscribed address.  I'm
> guessing someone just moderated the original through.
>
>

My apologies, it gave me the impression you were escallating an argument to roll trunk back 3 years.  

Unfortunately the tests can't prove the absence of errors, concurrency problems can lie dormant for years.  The tests passed previously with inadequate synchronization, it's plausible that client code could also have inadequate synchronization and experience issues.

There are a number of Jira's I need to follow up on, these known issues may be related to the random failures, one in particular explains how unsynchronized access is used to avoid deadlock:

River-145
River-348
River-258
River-140
River-113
River-43
River-37
River-30 (includes patch)

The tests can be run against previous releases to simulate an environment where only the test code has changed.



> Anyway, let's address one or two of your points...
>
> I see you writing inflammatory statements about my leadership skills and
> I think you're  upset because you think I was questioning the quality of
> your work. I understand.  You've put a lot of effort into the codebase.
>
> I feel sorry that you feel that way - it wasn't what I intended.
>
> Apache doesn't recognize any kind of a "project leader" position, and I
> don't pretend to hold any such influence over River.  I'm speaking as a
> committer and PMC member.  I certainly don't think I "hold the future of
> the project in my hands".  If anyone does hold individual control over
> the future of the project, then it doesn't qualify as an Apache project,
> and we need to remedy that.
>
> Really, what I'm trying to do is answer this question for myself - "Can
> I vote +1 on a release based on the trunk?".  There have been a lot of
> changes to the trunk code.  Yes, many that I don't understand.  I've
> done more management than you thnk.  I don't require that I understand
> everything.  That leads me to ask "How can I be confident about a
> release?"
>
> The best answer I have is to ask "does it pass the regression tests?".
> But that implies another question - "Do I trust the tests?"  And the
> answer to that is "currently, no, because from what I can see there have
> also been changes to the tests".
>
> I'm honestly and truly not passing judgement on the quality of the
> code.  I honestly don't know if it's good or bad.  I have to confess
> that, given that Jini was written as a top-level project at Sun,
> sponsored by Bill Joy, when Sun was at the top of its game, and the Jini
> project team was a "who's-who" of distributed computing pioneers, the
> idea that it's riddled with concurrency bugs surprises me.  But mainly,
> I'm still trying to answer that question - "How do I know if it's good?"
>
> Here's what I'm doing:
>
> - I'm attempting to run the tests from "tags/2.2.0" against the "2.2"
> branch.  When I have confidence in the "2.2" branch, I'll publish the
> results, ask anyone else who's interested to test it, and then call for
> a release on "2.2.1"
> - After that, the developers need to reach consensus about how to move
> forward.
>
> Cheers,
>
> Greg.
>
>
>
> > I think this is a deliberate attack on the project because you haven't
> > been following development in trunk and now you're scared because you
> > see changes you don't understand.
> >
> > I've been following your developments in surrogates, an impressive
> > amount of productivity.  Although I think you should consider
> > upgrading apache.commons vfs to version 2 before releasing.
> >
> > Open your mind and ask questions, the code isn't set in stone, you
> > have an obligation as project lead to encourage and nurture
> > development, not stifle it.
> >
> > You strike me as someone who's a very good programmer, but still
> > learning leadership because you lack faith in others and must do
> > everything yourself.  Remember I offered to assist with Surrogates,
> > but you wanted to work alone?
> >
> > You need to let go and give others a go too.
> >
> > How you handle this matter will be a test for your own personal
> > development and an opportunity to grow as a leader.
> >
> > You also hold the future of this project in your hands, so I hope you
> > find strength to let go.
> >
> > Regards,
> >
> > Peter.
> >
> > ----- Original message -----
> > >
> > > OK, so in my last message I talked about how (speaking only for
> > myself) I'm a
> > > little nervous about the state of the trunk.
> > >
> > > So what now?
> > >
> > > Problems we need to avoid in this discussion:
> > > -------------------------------------------------------------
> > >
> > > - Conflation of source tree structure issues with build tool
> > selection.
> > > - Conflation of Maven build, Maven as codebase provider (artifact
> > urls), and
> > > posting artifacts to Maven Central - Wish lists of pet features
> > > - Bruised egos and personal criticisms.
> > >
> > > Issues I see, in no particular order:
> > > ----------------------------------------------
> > > - We've done changes both to the test framework and the code, and
> > lots of them.
> > > We should do one or the other, or small amounts of coevolution, if
> > absolutely
> > > necessary. - Really, I'd like to see a completely separate
> > integration test, and
> > > have the TCK tests separated out again. - The source tree is
> > incomprehensible -
> > > The tests appear to be awfully sensitive to their environment.
> > Insofar as when
> > > I run them locally on an untouched source tree, I get 280 failures.
> > - There have
> > > been changes to class loading and security subsystems.  These
> > subsystems are
> > > core to Jini, and the changes were made to the existing source, so
> > there's no
> > > way to "opt-out" of the changes.  I'd like to see radical changes be
> > optional
> > > until proven in the field, where possible.  In the case of policy
> > providers and
> > > class loaders, that should be easy to do. - Similarly, it seems
> > there have been
> > > some changes to the JERI framework. - There are ".jar" files in our
> > repository.
> > > I'll stipulate that the licensing has been checked, but it smells
> > bad.
> > >
> > > Discussion
> > > -----------------
> > > I guess the biggest thing I'd like to see is stability in the test
> > framework.
> > > Perhaps it needs refactoring or reorganization, but if so, we need
> > to be very
> > > careful to separate it from changes to the core functionality.
> > >
> > > Next, I'd like for it to be easier to comprehend the source tree.  I
> > think a
> > > good way to do that is to separate out (carefully) the core Jini
> > package
> > > (basically the contents of jsk-platform.jar) and the service
> > implementations.
> > > There's no reason that we have to have one huge
> > everything-but-the-kitchen-sink
> > > distribution.  That's just a holdover from how Sun structured the
> > JTSK - It was
> > > literally a "starter kit".  To me it would be fine to have separate
> > deliverables
> > > for the platform and the services.
> > >
> > > While we're separating out the services, it might also be a decent
> > time to
> > > implement Maven-based builds if we think that's a good idea.  I'd
> > start with
> > > Reggie.  It would also be a good time to get rid of the
> > "com.sun.jini" packages.
> > >
> > > Aside:  I'm personally ambivalent on Maven (which is to say I'm
> > nowhere near as
> > > negative on it as I once was).  I do agree with Dennis, though, that
> > the jars
> > > and appropriate poms need to be published to Maven Central.  There's
> > no doubt
> > > that users will appreciate that.
> > >
> > > Once we have a stable set of regression tests, then OK, we could
> > think about
> > > improving performance or using Maven repositories as the codebase
> > server.
> > >
> > > I realize this won't be popular, but my gut feel is that we need to
> > step back to
> > > the 2.2 branch and retrace our steps a little, and go through the
> > evolution
> > > again in a more measured fashion.
> > >
> > > Proposal
> > > ------------
> > >
> > > 1 - Release version 2.2.1 from the 2.2 branch.
> > > 2 - Create a separate source tree for the test framework.  This
> > could come from
> > > the "qa_refactor" branch, but the goal should be to successfully
> > test the 2.2.1
> > > release.  Plus it should be a no-brainer to pull it down and run it
> > on a local
> > > machine. 3 - Release 2.2.2 from the pruned jtsk tree.  Release 1.0.0
> > of the test
> > > framework. 4 - Pull out the infrastructure service implementations
> > (Reggie,
> > > Outrigger, Norm, etc) from the core into separate products.  Release
> > 1.0.0 on
> > > each of them.  Release 2.2.3 from the pruned jtsk tree. 5 - Adopt a
> > fixed
> > > release cycle.  Not sure if it should be quarterly or biennial, or
> > whether it
> > > should be all products at once or staggered releases.  We'll need to
> > discuss. 6
> > > - Then we can start making changes if necessary to the individual
> > products.  And
> > > also try to deal with making it easier for new users to use the
> > technology.
> > >
> > > So there you go.  Opinions?
> > >
> > > Greg Trasuk.
> > >
> >
>

Re: Next steps after 2.2.1 release

[Tom Hobbs <tv...@googlemail.com>]

I'm not sure where I stand with regards to votes and what-not anymore, but
here's my opinion.  I think it's wise to do a quick release with the
minimum of changes in Right Now.  Particularly if those changes includes
the JDK7 fix.  Releasing as many changes as there are without more
consideration just feels too risky.  The greater need right now is to get
the bare minimum release out asap to fix some real-life issues.

That's not a comment on what I think the quality of all the trunk changes
are, it's just my gut feel.

To counter some of Greg's comments, if we trusted the old tests for
previous releases, then digging our way out of the hole is not hugely
difficult.  We just run the old tests against the new code to verify the
new code.  Then we run the new tests against an old release.  There will be
some additional due diligence needed to tie up loose ends and un-grey some
areas, but I think we can then have a good degree of confidence in both the
new tests and new code.

When that's done, I would suggest that it sounds like another release would
be due.  Then the source tree can be straightened out with the right bits
merged to the right branches and the right branches being created for the
right work streams.

I don't think that there is much value in debating the motives people have
had for the code they've written/changed.  As has been said before, we've
all got our own itches to scratch.  If there is a technical reason for
blocking some change then fine, but I don't believe that a lack of
benchmarks detailing some pain point is a reason to throw it out.
 Questioning the Why is often useful because it can aid a discussion and
guide us to what the real What should be, but I'm not sure that it will in
the case - then again, I've been wrong before...  So I think that it is a
good approach to modify the code so it follows the advice given in what
many would consider the "Concurrency Bible" - even if I can't prove that
the previous implementation was flawed in some way.

Dan has mentioned the policy (or lack of) with regards to what gets put
onto trunk and that is something that should really be discussed.  So some
questions;

- Is there a policy?
- Was it the right policy?
- Did we stick to the policy?
- What is a better policy?

Then everything else from Maven, to separate builds, to TaskManager
replacements (or not), to whatever else should just slot into place.

Lastly, I'm glad that the conversation has calmed down.  Thanks to both
Greg for not biting and Peter for responding in kind.  Your reactions to
what could have become a nasty situation speaks volumes to both of your
characters and that's A Good Thing.

Cheers,

Tom


On Mon, Apr 8, 2013 at 2:42 PM, Dan Creswell <da...@gmail.com> wrote:

> > This is an important issue to address.  I know a lot of people here
> >> probably don't participate on the Concurrency-interest mailing list that
> >> has a wide range of discussion about the JLS vs the JMM and what the JIT
> >> compilers actually do to code these days.
> >>
> > ...
> >
> > I used to be a concurrency expert, but have not been following the topic
> > recently. For practical Java coding, I have tended to follow the ideas
> > in Java Concurrency in Practice. Do any of the changes invalidate that
> > approach?
> >
> >
> No, they don't. The JMM hasn't really changed since the work Doug Lea did
> for Java 5 and beyond. What has changed over time is the amount typical
> JITs exploit the opportunities presented by the JMM for aggressive
> instruction re-ordering etc.
>
> If your code "does the right things" it'll be fine. It just potentially
> runs better (it could actually run worse in some cases). If you've
> misunderstood JMM or how it relates to JLS then you may have problems.
>

Re: Next steps after 2.2.1 release

[Dan Creswell <da...@gmail.com>]

> This is an important issue to address.  I know a lot of people here
>> probably don't participate on the Concurrency-interest mailing list that
>> has a wide range of discussion about the JLS vs the JMM and what the JIT
>> compilers actually do to code these days.
>>
> ...
>
> I used to be a concurrency expert, but have not been following the topic
> recently. For practical Java coding, I have tended to follow the ideas
> in Java Concurrency in Practice. Do any of the changes invalidate that
> approach?
>
>
No, they don't. The JMM hasn't really changed since the work Doug Lea did
for Java 5 and beyond. What has changed over time is the amount typical
JITs exploit the opportunities presented by the JMM for aggressive
instruction re-ordering etc.

If your code "does the right things" it'll be fine. It just potentially
runs better (it could actually run worse in some cases). If you've
misunderstood JMM or how it relates to JLS then you may have problems.

Re: Next steps after 2.2.1 release

[Patricia Shanahan <pa...@acm.org>]

On 4/8/2013 6:11 AM, Gregg Wonderly wrote:
> On 4/7/2013 7:03 PM, Greg Trasuk wrote:
>> I'm honestly and truly not passing judgement on the quality of the
>> code. I honestly don't know if it's good or bad. I have to confess
>> that, given that Jini was written as a top-level project at Sun,
>> sponsored by Bill Joy, when Sun was at the top of its game, and the
>> Jini project team was a "who's-who" of distributed computing pioneers,
>> the idea that it's riddled with concurrency bugs surprises me. But
>> mainly, I'm still trying to answer that question - "How do I know if
>> it's good?" Here's what I'm doing: - I'm attempting to run the tests
>> from "tags/2.2.0" against the "2.2" branch. When I have confidence in
>> the "2.2" branch, I'll publish the results, ask anyone else who's
>> interested to test it, and then call for a release on "2.2.1" - After
>> that, the developers need to reach consensus about how to move
>> forward. Cheers, Greg.
>
> This is an important issue to address.  I know a lot of people here
> probably don't participate on the Concurrency-interest mailing list that
> has a wide range of discussion about the JLS vs the JMM and what the JIT
> compilers actually do to code these days.
...

I used to be a concurrency expert, but have not been following the topic
recently. For practical Java coding, I have tended to follow the ideas
in Java Concurrency in Practice. Do any of the changes invalidate that
approach?

Patricia

Re: Next steps after 2.2.1 release

[Patricia Shanahan <pa...@acm.org>]

On 4/11/2013 4:15 AM, Peter Firmstone wrote:
...
> I know some would prefer me to prove something is broken before fixing
> it, providing tests that prove the failure, but this isn't an enterprise
> project and I lack the resources for such things, there's always the
> option of running a 2.2 maintenance branch for those who'd like to wait
> longer before upgrading.
...

I'm still trying to remember/find a place where there seemed to me to be 
a possible race condition.

Back when I was actively working on this, I considered trying to set up 
a test, and all I could think of was to put a moderately long 
Thread.sleep() call in the code where I thought there was a window. I 
was not able to produce a failing test.

Patricia

Re: Next steps after 2.2.1 release

[Peter Firmstone <ji...@zeus.net.au>]

Gregg,

Thanks again for your support.

I refactored LookupDiscovery and tidied up LookupLocatorDiscovery.

If you get some time, I could use a hand with other classes you've 
already fixed.

I'm working on MailboxImpl presently, there's some very dubious code, 
Threads being started from inside their constructors, called from static 
init methods from within MailboxImpl's constructor.

I know some would prefer me to prove something is broken before fixing 
it, providing tests that prove the failure, but this isn't an enterprise 
project and I lack the resources for such things, there's always the 
option of running a 2.2 maintenance branch for those who'd like to wait 
longer before upgrading.

Regards,

Peter.

On 11/04/2013 2:00 AM, Gregg Wonderly wrote:
> I just want to extend this conversation a bit by saying that nearly everything about River is "concurrently accessed".  There are, of course several places, where work is done by one thread, at a time, but new threads are created to do that work, and that means that "visibility" has to be considered.
>
> I won't say that every single field in every class in River needs to be final or volatile, but that should not be considered an extreme.  Specifically, you might see code execute just fine without appropriate concurrency design, and then it will suddenly break when a new optimization appears on the scene, reordering something under the covers and creating an intangible behavior.  Some "visibility bugs" might not ever manifest because of other "happens before" and "cache line sync" activities that happen implicitly based on the "current design" or "thread model".  We can "be happy" with "it ain't broke, so don't fix it", but I don't think that's very productive.
>
> I personally, have been beating on various parts of Jini in my "fork" because of completely unpredictable results in discovery and discovery management.  I've written, rewritten, debugged and stared at that code till I was blue in the face, because my ServiceUI desktop application just doesn't behave like it should.  Some of it is missing lifecycle management that was not in the original services, because System.registerShutdownHook() hasn't been used.  But other parts are these race conditions and thread scheduling overlaps (or underlaps) which keep discovery and notification from happening reliably.   There are lots of different reasons why people might not be "complaining" about this stuff, but I would contend that the fact that there are many examples of people forking and extending Jini, which to me, reflects the fact that there are things that aren't correct, or functional in the wild, and this causes them to jump over the cliff and never look back.
>
> We are at that point today, and Peter's continued slogging through the motions to track down and discover where the issues actually are, is an astronomical effort!  I have been very involved in several different, new work opportunities that have kept me from jumping in to participate in dealing with all of these issues, as I have really wanted to.
>
> Gregg Wonderly
>
> On Apr 8, 2013, at 3:19 PM, Peter<ji...@zeus.net.au>  wrote:
>
>> Thanks Gregg,
>>
>> You've hit the nail on the head, this is exactly the issue I'm having.
>>
>> So I've been fixing safe publication in constructors by making fields final or volatile and ensuring "this" doesn't escape, fixing synchronisation on collections etc during method calls.
>>
>> To fix deadlock, I investigate immutable non blocking data structures with volatile publication, if future state doesn't depend on previous state, if it does a CAS atomic reference can be used instead of volatile.
>>
>> Often i find synchronization is quite acceptable if it is limited in scope, if synchronized or holding a lock while a thread is executing outside your objects scope of control, that's when deadlock is more likely to occur.
>>
>> The polciy providers were deadlock prone, which is why they're mostly immutable non blocking now, any synchronization or locking is limited.
>>
>> I basically follow Doug Lea's concurrency in practise guidelines.
>>
>> For debugging I follow Cliff Click's reccommendations.
>>
>> Unfortunately fixing concurrency bugs means finding a trace of execution, identifying all classes and inspecting the code visually.  Findbugs identifies cases of inadequate sychronization using static analysis.
>>
>> Regards,
>>
>> Peter.
>>
>> ----- Original message -----
>>> On 4/7/2013 7:03 PM, Greg Trasuk wrote:
>>>> I'm honestly and truly not passing judgement on the quality of the code. I
>>>> honestly don't know if it's good or bad. I have to confess that, given that
>>>> Jini was written as a top-level project at Sun, sponsored by Bill Joy, when
>>>> Sun was at the top of its game, and the Jini project team was a "who's-who" of
>>>> distributed computing pioneers, the idea that it's riddled with concurrency
>>>> bugs surprises me. But mainly, I'm still trying to answer that question - "How
>>>> do I know if it's good?" Here's what I'm doing: - I'm attempting to run the
>>>> tests from "tags/2.2.0" against the "2.2" branch. When I have confidence in
>>>> the "2.2" branch, I'll publish the results, ask anyone else who's interested
>>>> to test it, and then call for a release on "2.2.1" - After that, the
>>>> developers need to reach consensus about how to move forward. Cheers, Greg.
>>> This is an important issue to address.  I know a lot of people here probably
>>> don't participate on the Concurrency-interest mailing list that has a wide range
>>> of discussion about the JLS vs the JMM and what the JIT compilers actually do to
>>> code these days.
>>>
>>> The number one issue that you need to understand, is that the optimizer is
>>> working against you more and more these days if you don't have JMM details
>>> exactly write.  Statements are being reordered more and more, including actual
>>> "assignments" which can expose uninitialized data items in "racy" concurrent
>>> code.  The latest example is the  Thread.setName()/Thread.getName() pair.  They
>>> are most likely always to be accessed by "other threads", yet there is no
>>> synchronization on them, including no "visibility" control with volatile even.
>>> What this means, is that if setName() and getName() are being called in a racy
>>> environment, the setName, will assign the array that is created to copy the
>>> characters into, before the arraycopy of the data occurs, potentially exposing
>>> an uninitialized name to getName().
>>>
>>> There are literally hundreds of places in the JDK that still have these kinds of
>>> races going on, and no one at Oracle, based on how people are acting, appears to
>>> be responsible for dealing with it. The Jini code, has many many of the same
>>> issues that just randomly appear in stress cases on "slower" or "faster"
>>> hardware, depending on the issue.
>>>
>>> When you haven't got sharing and visibility covered correctly, the JIT code
>>> rewrites can make execution order play a big part in conflating what you "see"
>>> happening verses what the "code" says, to you, should happen.
>>>
>>> There are some very simple things to get the JIT out of the picture.  One of
>>> these, is to actually open the source up in an IDE and declare every field
>>> final.  If that doesn't work due to 'mutation' of values, change those fields to
>>> 'volatile' so that it will compile again.    Then run your tests and you will now
>>> greatly diminish reordering and visibility issues so that you can just get to
>>> the simple "was it set correctly, before it was read" and "did we provide the
>>> correct atomicity for that update" kinds of questions that will help you
>>> understand things better when code is misbehaving.
>>>
>>> This is the kind of thing that Peter has been working through because the usage
>>> of the code in real life has not continued in the same way that it did when the
>>> code was written, and the JMM in JDK5 has literally broken so much software, all
>>> over the planet, that used to work quite well, because there wasn't a formal
>>> definition of "happens before".    Now that there is, the compiler optimizations
>>> are against you if you don't get it right.  The behaviors you will experience,
>>> because of reorderings that are targeted at all out performance (minimize
>>> traffic in and out of the CPU through memory subsystems), can create completely
>>> unexpected results.  Intra-thread semantics are kept correct, but inter-thread
>>> execution will just seem intangible because stuff will not be happening in the
>>> order the "code" says it should.
>>>
>>> Gregg Wonderly
>>>

Re: Next steps after 2.2.1 release

[Gregg Wonderly <ge...@cox.net>]

I just want to extend this conversation a bit by saying that nearly everything about River is "concurrently accessed".  There are, of course several places, where work is done by one thread, at a time, but new threads are created to do that work, and that means that "visibility" has to be considered.

I won't say that every single field in every class in River needs to be final or volatile, but that should not be considered an extreme.  Specifically, you might see code execute just fine without appropriate concurrency design, and then it will suddenly break when a new optimization appears on the scene, reordering something under the covers and creating an intangible behavior.  Some "visibility bugs" might not ever manifest because of other "happens before" and "cache line sync" activities that happen implicitly based on the "current design" or "thread model".  We can "be happy" with "it ain't broke, so don't fix it", but I don't think that's very productive.

I personally, have been beating on various parts of Jini in my "fork" because of completely unpredictable results in discovery and discovery management.  I've written, rewritten, debugged and stared at that code till I was blue in the face, because my ServiceUI desktop application just doesn't behave like it should.  Some of it is missing lifecycle management that was not in the original services, because System.registerShutdownHook() hasn't been used.  But other parts are these race conditions and thread scheduling overlaps (or underlaps) which keep discovery and notification from happening reliably.   There are lots of different reasons why people might not be "complaining" about this stuff, but I would contend that the fact that there are many examples of people forking and extending Jini, which to me, reflects the fact that there are things that aren't correct, or functional in the wild, and this causes them to jump over the cliff and never look back.

We are at that point today, and Peter's continued slogging through the motions to track down and discover where the issues actually are, is an astronomical effort!  I have been very involved in several different, new work opportunities that have kept me from jumping in to participate in dealing with all of these issues, as I have really wanted to.  

Gregg Wonderly

On Apr 8, 2013, at 3:19 PM, Peter <ji...@zeus.net.au> wrote:

> Thanks Gregg,
> 
> You've hit the nail on the head, this is exactly the issue I'm having.
> 
> So I've been fixing safe publication in constructors by making fields final or volatile and ensuring "this" doesn't escape, fixing synchronisation on collections etc during method calls.
> 
> To fix deadlock, I investigate immutable non blocking data structures with volatile publication, if future state doesn't depend on previous state, if it does a CAS atomic reference can be used instead of volatile.
> 
> Often i find synchronization is quite acceptable if it is limited in scope, if synchronized or holding a lock while a thread is executing outside your objects scope of control, that's when deadlock is more likely to occur.
> 
> The polciy providers were deadlock prone, which is why they're mostly immutable non blocking now, any synchronization or locking is limited.
> 
> I basically follow Doug Lea's concurrency in practise guidelines.
> 
> For debugging I follow Cliff Click's reccommendations.
> 
> Unfortunately fixing concurrency bugs means finding a trace of execution, identifying all classes and inspecting the code visually.  Findbugs identifies cases of inadequate sychronization using static analysis.
> 
> Regards,
> 
> Peter.
> 
> ----- Original message -----
>> On 4/7/2013 7:03 PM, Greg Trasuk wrote:
>>> I'm honestly and truly not passing judgement on the quality of the code. I
>>> honestly don't know if it's good or bad. I have to confess that, given that
>>> Jini was written as a top-level project at Sun, sponsored by Bill Joy, when
>>> Sun was at the top of its game, and the Jini project team was a "who's-who" of
>>> distributed computing pioneers, the idea that it's riddled with concurrency
>>> bugs surprises me. But mainly, I'm still trying to answer that question - "How
>>> do I know if it's good?" Here's what I'm doing: - I'm attempting to run the
>>> tests from "tags/2.2.0" against the "2.2" branch. When I have confidence in
>>> the "2.2" branch, I'll publish the results, ask anyone else who's interested
>>> to test it, and then call for a release on "2.2.1" - After that, the
>>> developers need to reach consensus about how to move forward. Cheers, Greg.
>> 
>> This is an important issue to address.  I know a lot of people here probably
>> don't participate on the Concurrency-interest mailing list that has a wide range
>> of discussion about the JLS vs the JMM and what the JIT compilers actually do to
>> code these days.
>> 
>> The number one issue that you need to understand, is that the optimizer is
>> working against you more and more these days if you don't have JMM details
>> exactly write.  Statements are being reordered more and more, including actual
>> "assignments" which can expose uninitialized data items in "racy" concurrent
>> code.  The latest example is the  Thread.setName()/Thread.getName() pair.  They
>> are most likely always to be accessed by "other threads", yet there is no
>> synchronization on them, including no "visibility" control with volatile even. 
>> What this means, is that if setName() and getName() are being called in a racy
>> environment, the setName, will assign the array that is created to copy the
>> characters into, before the arraycopy of the data occurs, potentially exposing
>> an uninitialized name to getName().
>> 
>> There are literally hundreds of places in the JDK that still have these kinds of
>> races going on, and no one at Oracle, based on how people are acting, appears to
>> be responsible for dealing with it. The Jini code, has many many of the same
>> issues that just randomly appear in stress cases on "slower" or "faster"
>> hardware, depending on the issue.
>> 
>> When you haven't got sharing and visibility covered correctly, the JIT code
>> rewrites can make execution order play a big part in conflating what you "see"
>> happening verses what the "code" says, to you, should happen.
>> 
>> There are some very simple things to get the JIT out of the picture.  One of
>> these, is to actually open the source up in an IDE and declare every field
>> final.  If that doesn't work due to 'mutation' of values, change those fields to
>> 'volatile' so that it will compile again.    Then run your tests and you will now
>> greatly diminish reordering and visibility issues so that you can just get to
>> the simple "was it set correctly, before it was read" and "did we provide the
>> correct atomicity for that update" kinds of questions that will help you
>> understand things better when code is misbehaving.
>> 
>> This is the kind of thing that Peter has been working through because the usage
>> of the code in real life has not continued in the same way that it did when the
>> code was written, and the JMM in JDK5 has literally broken so much software, all
>> over the planet, that used to work quite well, because there wasn't a formal
>> definition of "happens before".    Now that there is, the compiler optimizations
>> are against you if you don't get it right.  The behaviors you will experience,
>> because of reorderings that are targeted at all out performance (minimize
>> traffic in and out of the CPU through memory subsystems), can create completely
>> unexpected results.  Intra-thread semantics are kept correct, but inter-thread
>> execution will just seem intangible because stuff will not be happening in the
>> order the "code" says it should.
>> 
>> Gregg Wonderly
>> 
> 

Re: Next steps after 2.2.1 release

[Peter <ji...@zeus.net.au>]

Thanks Gregg,

You've hit the nail on the head, this is exactly the issue I'm having.

So I've been fixing safe publication in constructors by making fields final or volatile and ensuring "this" doesn't escape, fixing synchronisation on collections etc during method calls.

To fix deadlock, I investigate immutable non blocking data structures with volatile publication, if future state doesn't depend on previous state, if it does a CAS atomic reference can be used instead of volatile.

Often i find synchronization is quite acceptable if it is limited in scope, if synchronized or holding a lock while a thread is executing outside your objects scope of control, that's when deadlock is more likely to occur.

The polciy providers were deadlock prone, which is why they're mostly immutable non blocking now, any synchronization or locking is limited.

I basically follow Doug Lea's concurrency in practise guidelines.

For debugging I follow Cliff Click's reccommendations.

Unfortunately fixing concurrency bugs means finding a trace of execution, identifying all classes and inspecting the code visually.  Findbugs identifies cases of inadequate sychronization using static analysis.

Regards,

Peter.

----- Original message -----
> On 4/7/2013 7:03 PM, Greg Trasuk wrote:
> > I'm honestly and truly not passing judgement on the quality of the code. I
> > honestly don't know if it's good or bad. I have to confess that, given that
> > Jini was written as a top-level project at Sun, sponsored by Bill Joy, when
> > Sun was at the top of its game, and the Jini project team was a "who's-who" of
> > distributed computing pioneers, the idea that it's riddled with concurrency
> > bugs surprises me. But mainly, I'm still trying to answer that question - "How
> > do I know if it's good?" Here's what I'm doing: - I'm attempting to run the
> > tests from "tags/2.2.0" against the "2.2" branch. When I have confidence in
> > the "2.2" branch, I'll publish the results, ask anyone else who's interested
> > to test it, and then call for a release on "2.2.1" - After that, the
> > developers need to reach consensus about how to move forward. Cheers, Greg.
>
> This is an important issue to address.  I know a lot of people here probably
> don't participate on the Concurrency-interest mailing list that has a wide range
> of discussion about the JLS vs the JMM and what the JIT compilers actually do to
> code these days.
>
> The number one issue that you need to understand, is that the optimizer is
> working against you more and more these days if you don't have JMM details
> exactly write.  Statements are being reordered more and more, including actual
> "assignments" which can expose uninitialized data items in "racy" concurrent
> code.  The latest example is the  Thread.setName()/Thread.getName() pair.  They
> are most likely always to be accessed by "other threads", yet there is no
> synchronization on them, including no "visibility" control with volatile even. 
> What this means, is that if setName() and getName() are being called in a racy
> environment, the setName, will assign the array that is created to copy the
> characters into, before the arraycopy of the data occurs, potentially exposing
> an uninitialized name to getName().
>
> There are literally hundreds of places in the JDK that still have these kinds of
> races going on, and no one at Oracle, based on how people are acting, appears to
> be responsible for dealing with it. The Jini code, has many many of the same
> issues that just randomly appear in stress cases on "slower" or "faster"
> hardware, depending on the issue.
>
> When you haven't got sharing and visibility covered correctly, the JIT code
> rewrites can make execution order play a big part in conflating what you "see"
> happening verses what the "code" says, to you, should happen.
>
> There are some very simple things to get the JIT out of the picture.  One of
> these, is to actually open the source up in an IDE and declare every field
> final.  If that doesn't work due to 'mutation' of values, change those fields to
> 'volatile' so that it will compile again.    Then run your tests and you will now
> greatly diminish reordering and visibility issues so that you can just get to
> the simple "was it set correctly, before it was read" and "did we provide the
> correct atomicity for that update" kinds of questions that will help you
> understand things better when code is misbehaving.
>
> This is the kind of thing that Peter has been working through because the usage
> of the code in real life has not continued in the same way that it did when the
> code was written, and the JMM in JDK5 has literally broken so much software, all
> over the planet, that used to work quite well, because there wasn't a formal
> definition of "happens before".    Now that there is, the compiler optimizations
> are against you if you don't get it right.  The behaviors you will experience,
> because of reorderings that are targeted at all out performance (minimize
> traffic in and out of the CPU through memory subsystems), can create completely
> unexpected results.  Intra-thread semantics are kept correct, but inter-thread
> execution will just seem intangible because stuff will not be happening in the
> order the "code" says it should.
>
> Gregg Wonderly
>

Re: Next steps after 2.2.1 release

[Gregg Wonderly <gr...@wonderly.org>]

On 4/7/2013 7:03 PM, Greg Trasuk wrote:
> I'm honestly and truly not passing judgement on the quality of the code. I 
> honestly don't know if it's good or bad. I have to confess that, given that 
> Jini was written as a top-level project at Sun, sponsored by Bill Joy, when 
> Sun was at the top of its game, and the Jini project team was a "who's-who" of 
> distributed computing pioneers, the idea that it's riddled with concurrency 
> bugs surprises me. But mainly, I'm still trying to answer that question - "How 
> do I know if it's good?" Here's what I'm doing: - I'm attempting to run the 
> tests from "tags/2.2.0" against the "2.2" branch. When I have confidence in 
> the "2.2" branch, I'll publish the results, ask anyone else who's interested 
> to test it, and then call for a release on "2.2.1" - After that, the 
> developers need to reach consensus about how to move forward. Cheers, Greg.

This is an important issue to address.  I know a lot of people here probably 
don't participate on the Concurrency-interest mailing list that has a wide range 
of discussion about the JLS vs the JMM and what the JIT compilers actually do to 
code these days.

The number one issue that you need to understand, is that the optimizer is 
working against you more and more these days if you don't have JMM details 
exactly write.  Statements are being reordered more and more, including actual 
"assignments" which can expose uninitialized data items in "racy" concurrent 
code.  The latest example is the  Thread.setName()/Thread.getName() pair.  They 
are most likely always to be accessed by "other threads", yet there is no 
synchronization on them, including no "visibility" control with volatile even.  
What this means, is that if setName() and getName() are being called in a racy 
environment, the setName, will assign the array that is created to copy the 
characters into, before the arraycopy of the data occurs, potentially exposing 
an uninitialized name to getName().

There are literally hundreds of places in the JDK that still have these kinds of 
races going on, and no one at Oracle, based on how people are acting, appears to 
be responsible for dealing with it. The Jini code, has many many of the same 
issues that just randomly appear in stress cases on "slower" or "faster" 
hardware, depending on the issue.

When you haven't got sharing and visibility covered correctly, the JIT code 
rewrites can make execution order play a big part in conflating what you "see" 
happening verses what the "code" says, to you, should happen.

There are some very simple things to get the JIT out of the picture.  One of 
these, is to actually open the source up in an IDE and declare every field 
final.  If that doesn't work due to 'mutation' of values, change those fields to 
'volatile' so that it will compile again.   Then run your tests and you will now 
greatly diminish reordering and visibility issues so that you can just get to 
the simple "was it set correctly, before it was read" and "did we provide the 
correct atomicity for that update" kinds of questions that will help you 
understand things better when code is misbehaving.

This is the kind of thing that Peter has been working through because the usage 
of the code in real life has not continued in the same way that it did when the 
code was written, and the JMM in JDK5 has literally broken so much software, all 
over the planet, that used to work quite well, because there wasn't a formal 
definition of "happens before".   Now that there is, the compiler optimizations 
are against you if you don't get it right.  The behaviors you will experience, 
because of reorderings that are targeted at all out performance (minimize 
traffic in and out of the CPU through memory subsystems), can create completely 
unexpected results.  Intra-thread semantics are kept correct, but inter-thread 
execution will just seem intangible because stuff will not be happening in the 
order the "code" says it should.

Gregg Wonderly

Re: Next steps after 2.2.1 release

[Greg Trasuk <tr...@stratuscom.com>]

On Sun, 2013-04-07 at 17:54, Peter wrote:
> Greg, why have you repeated this message?
> 

First time I sent it was from the wrong email address, so it got hung up
in moderation.  I sent it again from my subscribed address.  I'm
guessing someone just moderated the original through.


Anyway, let's address one or two of your points...

I see you writing inflammatory statements about my leadership skills and
I think you're  upset because you think I was questioning the quality of
your work. I understand.  You've put a lot of effort into the codebase.

I feel sorry that you feel that way - it wasn't what I intended.

Apache doesn't recognize any kind of a "project leader" position, and I
don't pretend to hold any such influence over River.  I'm speaking as a
committer and PMC member.  I certainly don't think I "hold the future of
the project in my hands".  If anyone does hold individual control over
the future of the project, then it doesn't qualify as an Apache project,
and we need to remedy that.

Really, what I'm trying to do is answer this question for myself - "Can
I vote +1 on a release based on the trunk?".  There have been a lot of
changes to the trunk code.  Yes, many that I don't understand.  I've
done more management than you thnk.  I don't require that I understand
everything.  That leads me to ask "How can I be confident about a
release?"

The best answer I have is to ask "does it pass the regression tests?". 
But that implies another question - "Do I trust the tests?"  And the
answer to that is "currently, no, because from what I can see there have
also been changes to the tests".

I'm honestly and truly not passing judgement on the quality of the
code.  I honestly don't know if it's good or bad.  I have to confess
that, given that Jini was written as a top-level project at Sun,
sponsored by Bill Joy, when Sun was at the top of its game, and the Jini
project team was a "who's-who" of distributed computing pioneers, the
idea that it's riddled with concurrency bugs surprises me.  But mainly,
I'm still trying to answer that question - "How do I know if it's good?"

Here's what I'm doing:

- I'm attempting to run the tests from "tags/2.2.0" against the "2.2"
branch.  When I have confidence in the "2.2" branch, I'll publish the
results, ask anyone else who's interested to test it, and then call for
a release on "2.2.1"
- After that, the developers need to reach consensus about how to move
forward.

Cheers,

Greg.



> I think this is a deliberate attack on the project because you haven't
> been following development in trunk and now you're scared because you
> see changes you don't understand.
> 
> I've been following your developments in surrogates, an impressive
> amount of productivity.  Although I think you should consider
> upgrading apache.commons vfs to version 2 before releasing.
> 
> Open your mind and ask questions, the code isn't set in stone, you
> have an obligation as project lead to encourage and nurture
> development, not stifle it.
> 
> You strike me as someone who's a very good programmer, but still
> learning leadership because you lack faith in others and must do
> everything yourself.  Remember I offered to assist with Surrogates,
> but you wanted to work alone? 
> 
> You need to let go and give others a go too.
> 
> How you handle this matter will be a test for your own personal
> development and an opportunity to grow as a leader. 
> 
> You also hold the future of this project in your hands, so I hope you
> find strength to let go.
> 
> Regards,
> 
> Peter.
> 
> ----- Original message -----
> >
> > OK, so in my last message I talked about how (speaking only for
> myself) I'm a
> > little nervous about the state of the trunk.
> >
> > So what now? 
> >
> > Problems we need to avoid in this discussion:
> > -------------------------------------------------------------
> >
> > - Conflation of source tree structure issues with build tool
> selection.
> > - Conflation of Maven build, Maven as codebase provider (artifact
> urls), and
> > posting artifacts to Maven Central - Wish lists of pet features
> > - Bruised egos and personal criticisms.
> >
> > Issues I see, in no particular order:
> > ----------------------------------------------
> > - We've done changes both to the test framework and the code, and
> lots of them.
> > We should do one or the other, or small amounts of coevolution, if
> absolutely
> > necessary. - Really, I'd like to see a completely separate
> integration test, and
> > have the TCK tests separated out again. - The source tree is
> incomprehensible -
> > The tests appear to be awfully sensitive to their environment. 
> Insofar as when
> > I run them locally on an untouched source tree, I get 280 failures.
> - There have
> > been changes to class loading and security subsystems.  These
> subsystems are
> > core to Jini, and the changes were made to the existing source, so
> there's no
> > way to "opt-out" of the changes.  I'd like to see radical changes be
> optional
> > until proven in the field, where possible.  In the case of policy
> providers and
> > class loaders, that should be easy to do. - Similarly, it seems
> there have been
> > some changes to the JERI framework. - There are ".jar" files in our
> repository.
> > I'll stipulate that the licensing has been checked, but it smells
> bad.
> >
> > Discussion
> > -----------------
> > I guess the biggest thing I'd like to see is stability in the test
> framework.
> > Perhaps it needs refactoring or reorganization, but if so, we need
> to be very
> > careful to separate it from changes to the core functionality.
> >
> > Next, I'd like for it to be easier to comprehend the source tree.  I
> think a
> > good way to do that is to separate out (carefully) the core Jini
> package
> > (basically the contents of jsk-platform.jar) and the service
> implementations.
> > There's no reason that we have to have one huge
> everything-but-the-kitchen-sink
> > distribution.  That's just a holdover from how Sun structured the
> JTSK - It was
> > literally a "starter kit".  To me it would be fine to have separate
> deliverables
> > for the platform and the services.
> >
> > While we're separating out the services, it might also be a decent
> time to
> > implement Maven-based builds if we think that's a good idea.  I'd
> start with
> > Reggie.  It would also be a good time to get rid of the
> "com.sun.jini" packages.
> >
> > Aside:  I'm personally ambivalent on Maven (which is to say I'm
> nowhere near as
> > negative on it as I once was).  I do agree with Dennis, though, that
> the jars
> > and appropriate poms need to be published to Maven Central.  There's
> no doubt
> > that users will appreciate that.
> >
> > Once we have a stable set of regression tests, then OK, we could
> think about
> > improving performance or using Maven repositories as the codebase
> server.
> >
> > I realize this won't be popular, but my gut feel is that we need to
> step back to
> > the 2.2 branch and retrace our steps a little, and go through the
> evolution
> > again in a more measured fashion.
> >
> > Proposal
> > ------------
> >
> > 1 - Release version 2.2.1 from the 2.2 branch.
> > 2 - Create a separate source tree for the test framework.  This
> could come from
> > the "qa_refactor" branch, but the goal should be to successfully
> test the 2.2.1
> > release.  Plus it should be a no-brainer to pull it down and run it
> on a local
> > machine. 3 - Release 2.2.2 from the pruned jtsk tree.  Release 1.0.0
> of the test
> > framework. 4 - Pull out the infrastructure service implementations
> (Reggie,
> > Outrigger, Norm, etc) from the core into separate products.  Release
> 1.0.0 on
> > each of them.  Release 2.2.3 from the pruned jtsk tree. 5 - Adopt a
> fixed
> > release cycle.  Not sure if it should be quarterly or biennial, or
> whether it
> > should be all products at once or staggered releases.  We'll need to
> discuss. 6
> > - Then we can start making changes if necessary to the individual
> products.  And
> > also try to deal with making it easier for new users to use the
> technology.
> >
> > So there you go.  Opinions?
> >
> > Greg Trasuk.
> >
> 

Re: Next steps after 2.2.1 release

[Peter <ji...@zeus.net.au>]

Greg, why have you repeated this message?

I think this is a deliberate attack on the project because you haven't been following development in trunk and now you're scared because you see changes you don't understand.

I've been following your developments in surrogates, an impressive amount of productivity.  Although I think you should consider upgrading apache.commons vfs to version 2 before releasing.

Open your mind and ask questions, the code isn't set in stone, you have an obligation as project lead to encourage and nurture development, not stifle it.

You strike me as someone who's a very good programmer, but still learning leadership because you lack faith in others and must do everything yourself.  Remember I offered to assist with Surrogates, but you wanted to work alone? 

You need to let go and give others a go too.

How you handle this matter will be a test for your own personal development and an opportunity to grow as a leader. 

You also hold the future of this project in your hands, so I hope you find strength to let go.

Regards,

Peter.

----- Original message -----
>
> OK, so in my last message I talked about how (speaking only for myself) I'm a
> little nervous about the state of the trunk.
>
> So what now? 
>
> Problems we need to avoid in this discussion:
> -------------------------------------------------------------
>
> - Conflation of source tree structure issues with build tool selection.
> - Conflation of Maven build, Maven as codebase provider (artifact urls), and
> posting artifacts to Maven Central - Wish lists of pet features
> - Bruised egos and personal criticisms.
>
> Issues I see, in no particular order:
> ----------------------------------------------
> - We've done changes both to the test framework and the code, and lots of them.
> We should do one or the other, or small amounts of coevolution, if absolutely
> necessary. - Really, I'd like to see a completely separate integration test, and
> have the TCK tests separated out again. - The source tree is incomprehensible -
> The tests appear to be awfully sensitive to their environment.  Insofar as when
> I run them locally on an untouched source tree, I get 280 failures. - There have
> been changes to class loading and security subsystems.  These subsystems are
> core to Jini, and the changes were made to the existing source, so there's no
> way to "opt-out" of the changes.  I'd like to see radical changes be optional
> until proven in the field, where possible.  In the case of policy providers and
> class loaders, that should be easy to do. - Similarly, it seems there have been
> some changes to the JERI framework. - There are ".jar" files in our repository.
> I'll stipulate that the licensing has been checked, but it smells bad.
>
> Discussion
> -----------------
> I guess the biggest thing I'd like to see is stability in the test framework.
> Perhaps it needs refactoring or reorganization, but if so, we need to be very
> careful to separate it from changes to the core functionality.
>
> Next, I'd like for it to be easier to comprehend the source tree.  I think a
> good way to do that is to separate out (carefully) the core Jini package
> (basically the contents of jsk-platform.jar) and the service implementations.
> There's no reason that we have to have one huge everything-but-the-kitchen-sink
> distribution.  That's just a holdover from how Sun structured the JTSK - It was
> literally a "starter kit".  To me it would be fine to have separate deliverables
> for the platform and the services.
>
> While we're separating out the services, it might also be a decent time to
> implement Maven-based builds if we think that's a good idea.  I'd start with
> Reggie.  It would also be a good time to get rid of the "com.sun.jini" packages.
>
> Aside:  I'm personally ambivalent on Maven (which is to say I'm nowhere near as
> negative on it as I once was).  I do agree with Dennis, though, that the jars
> and appropriate poms need to be published to Maven Central.  There's no doubt
> that users will appreciate that.
>
> Once we have a stable set of regression tests, then OK, we could think about
> improving performance or using Maven repositories as the codebase server.
>
> I realize this won't be popular, but my gut feel is that we need to step back to
> the 2.2 branch and retrace our steps a little, and go through the evolution
> again in a more measured fashion.
>
> Proposal
> ------------
>
> 1 - Release version 2.2.1 from the 2.2 branch.
> 2 - Create a separate source tree for the test framework.  This could come from
> the "qa_refactor" branch, but the goal should be to successfully test the 2.2.1
> release.  Plus it should be a no-brainer to pull it down and run it on a local
> machine. 3 - Release 2.2.2 from the pruned jtsk tree.  Release 1.0.0 of the test
> framework. 4 - Pull out the infrastructure service implementations (Reggie,
> Outrigger, Norm, etc) from the core into separate products.  Release 1.0.0 on
> each of them.  Release 2.2.3 from the pruned jtsk tree. 5 - Adopt a fixed
> release cycle.  Not sure if it should be quarterly or biennial, or whether it
> should be all products at once or staggered releases.  We'll need to discuss. 6
> - Then we can start making changes if necessary to the individual products.  And
> also try to deal with making it easier for new users to use the technology.
>
> So there you go.  Opinions?
>
> Greg Trasuk.
>