Re: svn commit: r386058 [1/49] ...

[Leo Simons <ma...@leosimons.com>]

49 commit messages for a single commit! The continuous wash-in of
Really Big(tm) chunks of code scares me a little (even if its real cool)
 -- usually I make it policy to read every single line of code contributed
to a project for which I'm on the PMC but there's no chance in hell I'm
going to spend an entire weekend reading unit tests. Just keeping up amounts
to something close to a fulltime job. The "usual" "oversight" model that at
least some parts of the ASF are used to seems near-impossible to apply here.

Will all people able to read every line of code as it comes in please raise
their hands?

I'm thinking about how to make this stuff scale. Any ideas? The natural
tendency is to want to partition, but that way we lose the "many eyes"
advantages....

Anyway, just a random thought...

- Leo

Re: svn commit: r386058 [1/49] ...

[Stepan Mishura <st...@gmail.com>]

On 3/16/06, Tim Ellison wrote:
>
> Stepan Mishura wrote:
> > Leo,
> >
> > I agree with you that big commits are hard to review and to understand
> what
> > was changed. Why not to partition them?
>
> They are incoming contributions that were committed in bulk and
> reference the original JIRA.  We have all had a chance to look at the
> contributions and vote on them already.
>
> > At least I don't understand why
> > HARMONY-57 and HARMONY-88 were committed in one day.
>
> Why not?
>
> > Integrating these two
> > contributions of course changed the build: wow, the build now downloads
> > resources from the net and it failed to do this ...
>
> I agree, I don't like this either.  It will be fixed.
>
> > downloading them by
> > hands ... opsss, no property IdontHappyWithNetworkBuildUseLocalCopies
> ...
> > hacking the ant script .... wow, the script to run test trying to
> download
> > the same resources ... hacking the script ... hmmm, there are failed
> tests,
> > what is wrong? .... aha, old known problems ... well, are there any
> other
> > surprises? :-)
>
> C'mon Stepan, we are in a period of change while these bulk
> contributions are merged into the code base.  There are a few bumps in
> the road but they are being sorted out quickly, and the result is a much
> more functional runtime and lots more tests.  Please lend a hand rather
> than sniping from the sidelines :-(


Me shooting....? Oh, no ... please don't interpret my words in this way.

Thanks,
Stepan.

Regards,
> Tim
>
> > Thanks,
> > Stepan Mishura
> > Intel Middleware Products Division
> >
> >
> > On 3/16/06, Leo Simons <ma...@leosimons.com> wrote:
> >> 49 commit messages for a single commit! The continuous wash-in of
> >> Really Big(tm) chunks of code scares me a little (even if its real
> cool)
> >> -- usually I make it policy to read every single line of code
> contributed
> >> to a project for which I'm on the PMC but there's no chance in hell I'm
> >> going to spend an entire weekend reading unit tests. Just keeping up
> >> amounts
> >> to something close to a fulltime job. The "usual" "oversight" model
> that
> >> at
> >> least some parts of the ASF are used to seems near-impossible to apply
> >> here.
> >>
> >> Will all people able to read every line of code as it comes in please
> >> raise
> >> their hands?
> >>
> >> I'm thinking about how to make this stuff scale. Any ideas? The natural
> >> tendency is to want to partition, but that way we lose the "many eyes"
> >> advantages....
> >>
> >> Anyway, just a random thought...
> >>
> >> - Leo
> >>
> >
>
> --
>
> Tim Ellison (t.p.ellison@gmail.com)
> IBM Java technology centre, UK.
>



--
Thanks,
Stepan Mishura
Intel Middleware Products Division

Re: svn commit: r386058 [1/49] ...

[Tim Ellison <t....@gmail.com>]

Stepan Mishura wrote:
> Leo,
> 
> I agree with you that big commits are hard to review and to understand what
> was changed. Why not to partition them?

They are incoming contributions that were committed in bulk and
reference the original JIRA.  We have all had a chance to look at the
contributions and vote on them already.

> At least I don't understand why
> HARMONY-57 and HARMONY-88 were committed in one day. 

Why not?

> Integrating these two
> contributions of course changed the build: wow, the build now downloads
> resources from the net and it failed to do this ...

I agree, I don't like this either.  It will be fixed.

> downloading them by
> hands ... opsss, no property IdontHappyWithNetworkBuildUseLocalCopies ...
> hacking the ant script .... wow, the script to run test trying to download
> the same resources ... hacking the script ... hmmm, there are failed tests,
> what is wrong? .... aha, old known problems ... well, are there any other
> surprises? :-)

C'mon Stepan, we are in a period of change while these bulk
contributions are merged into the code base.  There are a few bumps in
the road but they are being sorted out quickly, and the result is a much
more functional runtime and lots more tests.  Please lend a hand rather
than sniping from the sidelines :-(

Regards,
Tim

> Thanks,
> Stepan Mishura
> Intel Middleware Products Division
> 
> 
> On 3/16/06, Leo Simons <ma...@leosimons.com> wrote:
>> 49 commit messages for a single commit! The continuous wash-in of
>> Really Big(tm) chunks of code scares me a little (even if its real cool)
>> -- usually I make it policy to read every single line of code contributed
>> to a project for which I'm on the PMC but there's no chance in hell I'm
>> going to spend an entire weekend reading unit tests. Just keeping up
>> amounts
>> to something close to a fulltime job. The "usual" "oversight" model that
>> at
>> least some parts of the ASF are used to seems near-impossible to apply
>> here.
>>
>> Will all people able to read every line of code as it comes in please
>> raise
>> their hands?
>>
>> I'm thinking about how to make this stuff scale. Any ideas? The natural
>> tendency is to want to partition, but that way we lose the "many eyes"
>> advantages....
>>
>> Anyway, just a random thought...
>>
>> - Leo
>>
> 

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

Re: svn commit: r386058 [1/49] ...

[Stepan Mishura <st...@gmail.com>]

On 3/16/06, Mark Hindess wrote:
>
> On 3/16/06, Stepan Mishura wrote:
> > Hi Mark,
> >
> > >
> > >I will submit patches to improve the download process for the few
> > >people that don't have (transparent proxy) internet access.
> > >
> >
> > Please add property IdontHappyWithNetworkBuildUseLocalCopies :-)
>
> See thread "svn commit: r38617" for a brief outline of what I intend
> or just give me an hour or so to submit something to JIRA. ;-)
>
> Rest assured my proposed solution will only attempt network access
> *if* the files are missing.


Agree.

> >
> > >Can you provide more details of the test failures?  I only have one
> > >which only affects the eclipse compiler which (probably correctly)
> > >doesn't like the non-ASCII characters in the URITest.java source.
> > >
> >
> > The problem is signed BC provider jar. May be it makes sense to update
> the
> > ant script to download BC provider's jar, remove signature and copy
> unsigned
> > jar file to deploy/jre/lib/ext.
>
> I'm surprised the old tests worked for you.


Old tests don't depend on this jar except security and x-net. Currently in
x-net module such tests are excluded and the ant script for security module
adds explicitly provider's jar to the bootclasspath as work around.

We don't currently automate that at all. Personally, I'm copying an
> unsigned bcprov.jar in to my tree between build and test.  I assumed
> this had already been covered on the list and that others were doing
> something similar already.
>
> I created the unsigned jar manually using:
>
> zip -d bcprov.jar META-INF/BCKEY.SF
>
> I'd need to check to see if this could be acheived using the ant zip task.
>
> It would be good to automate the download and removal of the key.  I
> wonder if we are allowed to do this I understand that we have to be
> careful what we include in the default build - will need to check the
> license.


If license forbids automating this process what about not removing unsigned
jar during clean up? So unsigning by hands will be done only once.

Don't expect this in my first attempt at cleaning up the depends process,
> but it
> is certainly something we need to work on.
>
> > Mark, you don't need to excuse for your work - everybody does mistakes.
> You
> > did great job with contributions integration! I only worried about how
> to
> > make process of tracking commits to repository more transparent and
> easy.
>
> I'm not sure it's ever going to be easy to handle commits of
> contributions.  Even for a committer I still feel that the approach of
> checking in and then fixing is the approach that gives the most
> clarity with respect to provenance.



Agree with small addition - new code should try gently expend existing
codebase rather then change everything dramatically.

The only exception I made to this was renaming tests because I know
> how much of a headache renaming can be ... specifically how patches
> get out of date when things move around before they get committed.



Yes, it is problem. May be it makes sense to specify dependencies between
patches? JIRA allows this. So if one patch makes another patch out of date
then JIRA let committer to know and she/he will try to apply patches in the
right order.

Thanks, Stepan.



Regards,
> Mark.
>
> --
> Mark Hindess <ma...@googlemail.com>
> IBM Java Technology Centre, UK.
>



--
Thanks,
Stepan Mishura
Intel Middleware Products Division

Re: svn commit: r386058 [1/49] ...

[Mark Hindess <ma...@googlemail.com>]

On 3/16/06, Stepan Mishura <st...@gmail.com> wrote:
> Hi Mark,
>
> >
> >I will submit patches to improve the download process for the few
> >people that don't have (transparent proxy) internet access.
> >
>
> Please add property IdontHappyWithNetworkBuildUseLocalCopies :-)

See thread "svn commit: r38617" for a brief outline of what I intend
or just give me an hour or so to submit something to JIRA. ;-)

Rest assured my proposed solution will only attempt network access
*if* the files are missing.

> >
> >Can you provide more details of the test failures?  I only have one
> >which only affects the eclipse compiler which (probably correctly)
> >doesn't like the non-ASCII characters in the URITest.java source.
> >
>
> The problem is signed BC provider jar. May be it makes sense to update the
> ant script to download BC provider's jar, remove signature and copy unsigned
> jar file to deploy/jre/lib/ext.

I'm surprised the old tests worked for you.

We don't currently automate that at all. Personally, I'm copying an
unsigned bcprov.jar in to my tree between build and test.  I assumed
this had already been covered on the list and that others were doing
something similar already.

I created the unsigned jar manually using:

  zip -d bcprov.jar META-INF/BCKEY.SF

I'd need to check to see if this could be acheived using the ant zip task.

It would be good to automate the download and removal of the key.  I
wonder if we are allowed to do this I understand that we have to be
careful what we include in the default build - will need to check the
license.

Don't expect this in my first attempt at cleaning up the depends process, but it
is certainly something we need to work on.

> Mark, you don't need to excuse for your work - everybody does mistakes. You
> did great job with contributions integration! I only worried about how to
> make process of tracking commits to repository more transparent and easy.

I'm not sure it's ever going to be easy to handle commits of
contributions.  Even for a committer I still feel that the approach of
checking in and then fixing is the approach that gives the most
clarity with respect to provenance.

The only exception I made to this was renaming tests because I know
how much of a headache renaming can be ... specifically how patches
get out of date when things move around before they get committed.

Regards,
 Mark.

--
Mark Hindess <ma...@googlemail.com>
IBM Java Technology Centre, UK.

Re: svn commit: r386058 [1/49] ...

[Stepan Mishura <st...@gmail.com>]

Hi Mark,

>
>I will submit patches to improve the download process for the few
>people that don't have (transparent proxy) internet access.
>

Please add property IdontHappyWithNetworkBuildUseLocalCopies :-)

>
>Can you provide more details of the test failures?  I only have one
>which only affects the eclipse compiler which (probably correctly)
>doesn't like the non-ASCII characters in the URITest.java source.
>

The problem is signed BC provider jar. May be it makes sense to update the
ant script to download BC provider's jar, remove signature and copy unsigned
jar file to deploy/jre/lib/ext.

>
>I suspect the old known problems you refer to are the test server
>issues?  Again, there was nothing I could do about these without
>adding IP to the integration process.
>I did however specifically not configure the test servers on my local
>machine and excluded all the tests that fail.  I felt this would be a
>reasonable compromise.  The alternative would have been excluding all
>the tests until everyone was happy.  But it would be a shame to take
>that route when there are thousands of working tests and likely less
>than 100 failures.
>

I'm OK with excluding failing tests until we work out acceptable solution.

>
>I was only trying to help our over-worked committers.  I apologies if
>I made mistakes.  I never claimed it would be perfect.  I'd be very
>happy to accept constructive comments on how I could have done it
>better (without adding extra IP and giving Geir headaches).
>

Mark, you don't need to excuse for your work - everybody does mistakes. You
did great job with contributions integration! I only worried about how to
make process of tracking commits to repository more transparent and easy.

Thanks,
Stepan



On 3/16/06, Mark Hindess  wrote:
>
> On 3/16/06, Stepan Mishura  wrote:
> > Leo,
> >
> > I agree with you that big commits are hard to review and to understand
> what
> > was changed. Why not to partition them? At least I don't understand why
> > HARMONY-57 and HARMONY-88 were committed in one day. Integrating
> > these two contributions of course changed the build: wow, the build now
> > downloads resources from the net and it failed to do this ...
> downloading them
> > by hands ... opsss, no property IdontHappyWithNetworkBuildUseLocalCopies
> ...
> > hacking the ant script .... wow, the script to run test trying to
> download
> > the same resources ... hacking the script ... hmmm, there are failed
> tests,
> > what is wrong? .... aha, old known problems ... well, are there any
> other
> > surprises? :-)
>
> Hi Stepan,
>
> As for the download issues, I've already mentioned on another thread
> that I was aware of this and have a plan to fix it.
>
> I wasn't in a position to do much about this in the script I submitted
> because Geir had already expressed concern over provenance issues and
> I went to some length to ensure I avoided adding anything new in the
> process.  (Which is why for instance I used sed to create the module
> ant files rather than including them in the patch - the latter would
> be easier but with the former it is clear I'm not contribution IP to
> the process.)
>
> I will submit patches to improve the download process for the few
> people that don't have (transparent proxy) internet access.
>
> Can you provide more details of the test failures?  I only have one
> which only affects the eclipse compiler which (probably correctly)
> doesn't like the non-ASCII characters in the URITest.java source.
>
> I suspect the old known problems you refer to are the test server
> issues?  Again, there was nothing I could do about these without
> adding IP to the integration process.
> I did however specifically not configure the test servers on my local
> machine and excluded all the tests that fail.  I felt this would be a
> reasonable compromise.  The alternative would have been excluding all
> the tests until everyone was happy.  But it would be a shame to take
> that route when there are thousands of working tests and likely less
> than 100 failures.
>
> The build machine I tested on has a very strict firewall, and the only
> additional rule I added to get the build to work was to allow outbound
> HTTP to ibilbio.
>
> I was only trying to help our over-worked committers.  I apologies if
> I made mistakes.  I never claimed it would be perfect.  I'd be very
> happy to accept constructive comments on how I could have done it
> better (without adding extra IP and giving Geir headaches).
>
> Regards,
> Mark.
>



--
Thanks,
Stepan Mishura
Intel Middleware Products Division

Re: svn commit: r386058 [1/49] ...

[Mark Hindess <ma...@googlemail.com>]

On 3/16/06, Stepan Mishura <st...@gmail.com> wrote:
> Leo,
>
> I agree with you that big commits are hard to review and to understand what
> was changed. Why not to partition them? At least I don't understand why
> HARMONY-57 and HARMONY-88 were committed in one day. Integrating
> these two contributions of course changed the build: wow, the build now
> downloads resources from the net and it failed to do this ... downloading them
> by hands ... opsss, no property IdontHappyWithNetworkBuildUseLocalCopies ...
> hacking the ant script .... wow, the script to run test trying to download
> the same resources ... hacking the script ... hmmm, there are failed tests,
> what is wrong? .... aha, old known problems ... well, are there any other
> surprises? :-)

Hi Stepan,

As for the download issues, I've already mentioned on another thread
that I was aware of this and have a plan to fix it.

I wasn't in a position to do much about this in the script I submitted
because Geir had already expressed concern over provenance issues and
I went to some length to ensure I avoided adding anything new in the
process.  (Which is why for instance I used sed to create the module
ant files rather than including them in the patch - the latter would
be easier but with the former it is clear I'm not contribution IP to
the process.)

I will submit patches to improve the download process for the few
people that don't have (transparent proxy) internet access.

Can you provide more details of the test failures?  I only have one
which only affects the eclipse compiler which (probably correctly)
doesn't like the non-ASCII characters in the URITest.java source.

I suspect the old known problems you refer to are the test server
issues?  Again, there was nothing I could do about these without
adding IP to the integration process.
I did however specifically not configure the test servers on my local
machine and excluded all the tests that fail.  I felt this would be a
reasonable compromise.  The alternative would have been excluding all
the tests until everyone was happy.  But it would be a shame to take
that route when there are thousands of working tests and likely less
than 100 failures.

The build machine I tested on has a very strict firewall, and the only
additional rule I added to get the build to work was to allow outbound
HTTP to ibilbio.

I was only trying to help our over-worked committers.  I apologies if
I made mistakes.  I never claimed it would be perfect.  I'd be very
happy to accept constructive comments on how I could have done it
better (without adding extra IP and giving Geir headaches).

Regards,
 Mark.

Re: svn commit: r386058 [1/49] ...

[Stepan Mishura <st...@gmail.com>]

Leo,

I agree with you that big commits are hard to review and to understand what
was changed. Why not to partition them? At least I don't understand why
HARMONY-57 and HARMONY-88 were committed in one day. Integrating these two
contributions of course changed the build: wow, the build now downloads
resources from the net and it failed to do this ... downloading them by
hands ... opsss, no property IdontHappyWithNetworkBuildUseLocalCopies ...
hacking the ant script .... wow, the script to run test trying to download
the same resources ... hacking the script ... hmmm, there are failed tests,
what is wrong? .... aha, old known problems ... well, are there any other
surprises? :-)

Thanks,
Stepan Mishura
Intel Middleware Products Division


On 3/16/06, Leo Simons <ma...@leosimons.com> wrote:
>
> 49 commit messages for a single commit! The continuous wash-in of
> Really Big(tm) chunks of code scares me a little (even if its real cool)
> -- usually I make it policy to read every single line of code contributed
> to a project for which I'm on the PMC but there's no chance in hell I'm
> going to spend an entire weekend reading unit tests. Just keeping up
> amounts
> to something close to a fulltime job. The "usual" "oversight" model that
> at
> least some parts of the ASF are used to seems near-impossible to apply
> here.
>
> Will all people able to read every line of code as it comes in please
> raise
> their hands?
>
> I'm thinking about how to make this stuff scale. Any ideas? The natural
> tendency is to want to partition, but that way we lose the "many eyes"
> advantages....
>
> Anyway, just a random thought...
>
> - Leo
>