[RT] a strategy for better gump action/reaction analysis

[Stefano Mazzocchi <st...@apache.org>]

"gump action/reaction analysis" could be otherwise named "blames" or 
"nags" but I like to think that if I break a build down 3 layers of 
dependencies, I'm not really screwing up, but merely I need to be 
advised or the 'reaction' of my 'action'.

I have already stated that finding the exact action (commit) that 
created a particular reaction (broken build) is, in general, very 
computationally intensive, but this is so only if we have to regenerate 
the state of the build without any sort of caching.

Our "jar archive" is an history of all the build-produced artifacts.

Now, I wonder, is this really what we want?

If left alone, this archive will slowly saturate all existing disk 
space. And pretty fast.

Do we really need to save *all* the states?

I think we just need one per project: the last successful one, tagged 
with the day of the checkout (or the SVN revision number, whichever you 
want)

Gump should remove the previous ones, save space and make things more 
certain.

                                  - o -

So, assuming that gump was working like that, how would a better (but 
still simple) algorithm work like?

Say we have project B that depends on project A.

Both projects compiled yesterday. We have artifacts from both in the 
archive.

Now, B doesn't build while A builds.

Whose fault is that? There is not enough information in a single build 
to prove that, so we have to build again, but changing the constrains.

If A builds but B doesn't, there are two possibilies:

  1) B broke itself
  2) A broke B

Since we know that B-yesterda built against A-yesterday, than the result 
of the B-today against A-yesterday will tell us if it's 1) or 2)

Ok, with two projects is easy, what about three?

  C -> B -> A

worked yesterday, but today C is broken and B and A work. Who broke the 
build?

First of all, is it safe to assume that since A is shielded by B and 
both B and A work, A has no influence on C?

If so, we just apply the same algorithm for [C -> B].

thoughts?

-- 
Stefano.

Re: [RT] a strategy for better gump action/reaction analysis

[Stefan Bodewig <bo...@apache.org>]

On Fri, 13 Aug 2004, Stefano Mazzocchi <st...@apache.org> wrote:

> Ok, with two projects is easy, what about three?
> 
>   C -> B -> A
> 
> worked yesterday, but today C is broken and B and A work. Who broke
> the build?
> 
> First of all, is it safe to assume that since A is shielded by B and
> both B and A work, A has no influence on C?

No.

B and C could depend on very different parts of A, even if the
dependency is no direct dependency.  It's rather common that B will
only need stable interfaces of A to compile.  If only the
implementation of A changes, B won't notice it.  If the build process
of C now touches a bug in A's changed implementation, C will break and
it is A's fault.

Consider A = commons-logging, B = httpclient and C's build involves
running httpclient.  C will not depend on commons-logging explicitly,
but inherit it from httpclient.  Now somebody accidentially commits a
change to commons-logging that causes a NullPointerException at
runtime ...

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org

Re: [RT] a strategy for better gump action/reaction analysis

["Adam R. B. Jack" <aj...@apache.org>]

> Do we really need to save *all* the states?
>
> I think we just need one per project: the last successful one, tagged with 
> the day of the checkout (or the SVN revision number, whichever you want)

When we copy from the SVN or CVS areas (from server) to the Gump work area, we detect if any files have changed (server side). As such, we could store (for as long as sensible) the last N changed outputs. [We only know modules changes, so each project is marked as change when a module changes. This is bad for jakarta-commons, but tolerable elsewhere, IMHO.]

> Gump should remove the previous ones, save space and make things more 
> certain.

Yup (in general).

> If so, we just apply the same algorithm for [C -> B].
>
> Thoughts?

Why not? If (say, as now) 70+% of the Gump tree is building, it is only the 'Gump high water mark' (http://brutus.apache.org/gump/public/project_todos.html) that we need to do this for. Gump cycles on build (during successes) are almost boring; the paint has dried but we are still looking. Spending cycles on the failures (however expensive) on new members of the Gump tide line, seems very worth while to me.

I am sure there are cases (i.e. both changed) when the combinatorial won't give us an answer, but hey -- in those case (on first ever event) -- why not notify all players. Gump ought work upstream, not just downstream, IMHO.

I like this, I hope we get there soon with it. Personally, I also thing that it is time to start parsing the ant output (like Leo mentioned before) and looking for clues. We ought be able to take a compile error, detect  the package, calcualte what project that is from, and target that one. I believe that is an avenue to explore soon...

regards,

Adam

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org

Re: [RT] a strategy for better gump action/reaction analysis

[Niclas Hedhman <ni...@hedhman.org>]

On Saturday 14 August 2004 08:39, Adam R. B. Jack wrote:

> BTW: We have the ability to do this cross-post notification only upon the
> first occurence of the event, and from then on 'nag' the breaking product
> ('cos it is the one that owns the issue, IMHO).

Example;
BCEL made an incompatible change in a method signature (now throwing a checked 
Exception), which sits in the CVS tree, but is not in the released package.
Excalibur used that method, and broke.
Why is Excalibur the 'owner of the issue'?? BCEL is the one who should fix it, 
either by making a 6.0 release, or make the change in a compatible fashion.

Cheers
Niclas
-- 
   +------//-------------------+
  / http://www.bali.ac        /
 / http://niclas.hedhman.org / 
+------//-------------------+


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org

Re: [RT] a strategy for better gump action/reaction analysis

["Adam R. B. Jack" <aj...@apache.org>]

>>> If A builds but B doesn't, there are two possibilies:
>>> 
>>>  1) B broke itself
>>>  2) A broke B
>> 
>> 
>> A breaking B doesn't imply fault. 
>
Agreed, and we just need to remove the 'blame' aspect, and consider notification, an 'FYI' style communication.

>
>> Counterexample: there were cases where log4j had deprecated an interface 
>> for literally YEARS (I'm not exagerating here), and when they removed it, 
>> builds broke.

Good example. Following it on, I recall Ceki didn't seem to mind knowing about the 'user' of log4j that had problems. Sure he might berate them for not getting onboard sooner, but he was willing to help them move. As I recall he did also revert on change, based off some of the communications threads/analyses that followed. The 'practical/real-world interface' (both parties & what is being used) had an event, that is what Gump is looking into. I am starting to wonder if we can consider the usage interface (dependency, but it's implications -- what classes/methods) as an entity w/ community ownership.

> Right. Log4j did an action that caused a reaction. Now, this is not log4j's 
> fault but a communication that needs to happen.

BTW: We have the ability to do this cross-post notification only upon the first occurence of the event, and from then on 'nag' the breaking product ('cos it is the one that owns the issue, IMHO).

BTW: I think Gump needs to integrate with JIRA and/or others. Allow a human to press a button to assign an issue id to an event. Folks can register with the tracker if they care to know about updates, etc.  Gump could attempt to emualte such tracking (and that might be wise, or folks might need to use N differnt trackers for N different dependees) but I'd like to consider starting with JIRA integration.

> wait, wait.
>
> if C depends on B which, in turn, depends on A, but C and A have no direct 
> dependencies and B compiles fine (which means that A compiled fine too), what 
> is going to happen?

Plenty of A can bleed into C, especially behaviours at runtime. I think it is likely the unusual case where there is no direct dependency.

We don't need exactly science here, we need community science...

regards,

Adam

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org

Re: [RT] a strategy for better gump action/reaction analysis

[Stefano Mazzocchi <st...@apache.org>]

Sam Ruby wrote:
> Stefano Mazzocchi wrote:
> 
>> Both projects compiled yesterday. We have artifacts from both in the 
>> archive.
> 
> 
> If you can do "cvs update -D", you don't need artifacts.  I used to 
> routinely be able to isolate the individual commit that caused a given 
> failure by one by one rolling back projects, until the failure went 
> away, and then doing a binomial search over the time range to isolate 
> the individual commit.

Artifacts reduce the build time. It's like a cache.

>> If A builds but B doesn't, there are two possibilies:
>>
>>  1) B broke itself
>>  2) A broke B
> 
> 
> A breaking B doesn't imply fault.  

No, but my suggestion is to move away from the psychology of "fault -> 
nag" and move into the 'action -> reaction' one.

> Counterexample: there were cases 
> where log4j had deprecated an interface for literally YEARS (I'm not 
> exagerating here), and when they removed it, builds broke.

Right. Log4j did an action that caused a reaction. Now, this is not 
log4j's fault but a communication that needs to happen.

> Nagging on deprecation warnings is difficult.  When the deprecation is 
> initially inserted into the development stream, there has been no 
> release, and projects are typically (and rightfully) reluctant to step 
> up to a daily build.

We should not indicate "who is right and who is wrong", but just 
indicate that "this action, made by this person at this particular time 
on this particular project" created a reaction on this other project 
that depends on it.

Both actor and actee (reactor?) are sent an email and you can be sure 
this will generate some communication.

>> Since we know that B-yesterda built against A-yesterday, than the 
>> result of the B-today against A-yesterday will tell us if it's 1) or 2)
>>
>> Ok, with two projects is easy, what about three?
>>
>>  C -> B -> A
>>
>> worked yesterday, but today C is broken and B and A work. Who broke 
>> the build?
>>
>> First of all, is it safe to assume that since A is shielded by B and 
>> both B and A work, A has no influence on C?
> 
> 
> What matters is the jars that are used to build C.  If A is visible at 
> all, then it it relevant.  It might not be directly involved at compile 
> time, but it could be relevant at runtime.  Furthermore, compile time 
> often involves things like XSLT translations, so the distinction between 
> compile time and runtime is not necessary pertinent.
> 
> I remember a Xalan change that broke FOP's build for this reason.

wait, wait.

if C depends on B which, in turn, depends on A, but C and A have no 
direct dependencies and B compiles fine (which means that A compiled 
fine too), what is going to happen?


-- 
Stefano.

Re: [RT] a strategy for better gump action/reaction analysis

[Sam Ruby <ru...@apache.org>]

Stefano Mazzocchi wrote:

> Both projects compiled yesterday. We have artifacts from both in the 
> archive.

If you can do "cvs update -D", you don't need artifacts.  I used to 
routinely be able to isolate the individual commit that caused a given 
failure by one by one rolling back projects, until the failure went 
away, and then doing a binomial search over the time range to isolate 
the individual commit.

> If A builds but B doesn't, there are two possibilies:
> 
>  1) B broke itself
>  2) A broke B

A breaking B doesn't imply fault.  Counterexample: there were cases 
where log4j had deprecated an interface for literally YEARS (I'm not 
exagerating here), and when they removed it, builds broke.

Nagging on deprecation warnings is difficult.  When the deprecation is 
initially inserted into the development stream, there has been no 
release, and projects are typically (and rightfully) reluctant to step 
up to a daily build.

> Since we know that B-yesterda built against A-yesterday, than the result 
> of the B-today against A-yesterday will tell us if it's 1) or 2)
> 
> Ok, with two projects is easy, what about three?
> 
>  C -> B -> A
> 
> worked yesterday, but today C is broken and B and A work. Who broke the 
> build?
> 
> First of all, is it safe to assume that since A is shielded by B and 
> both B and A work, A has no influence on C?

What matters is the jars that are used to build C.  If A is visible at 
all, then it it relevant.  It might not be directly involved at compile 
time, but it could be relevant at runtime.  Furthermore, compile time 
often involves things like XSLT translations, so the distinction between 
compile time and runtime is not necessary pertinent.

I remember a Xalan change that broke FOP's build for this reason.

- Sam Ruby

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org

Re: [RT] a strategy for better gump action/reaction analysis

[Nick Chalko <ni...@chalko.com>]

Stefano Mazzocchi wrote:

> "gump action/reaction analysis" could be otherwise named "blames" or 
> "nags" but I like to think that if I break a build down 3 layers of 
> dependencies, I'm not really screwing up, but merely I need to be 
> advised or the 'reaction' of my 'action'.
>
> I have already stated that finding the exact action (commit) that 
> created a particular reaction (broken build) is, in general, very 
> computationally intensive, but this is so only if we have to 
> regenerate the state of the build without any sort of caching.
>
> Our "jar archive" is an history of all the build-produced artifacts.
>
> Now, I wonder, is this really what we want?
>
> If left alone, this archive will slowly saturate all existing disk 
> space. And pretty fast.
>
> Do we really need to save *all* the states?
>
> I think we just need one per project: the last successful one, tagged 
> with the day of the checkout (or the SVN revision number, whichever 
> you want)
>
> Gump should remove the previous ones, save space and make things more 
> certain.
>
+1

>                                  - o -
>
> So, assuming that gump was working like that, how would a better (but 
> still simple) algorithm work like?
>
> Say we have project B that depends on project A.
>
> Both projects compiled yesterday. We have artifacts from both in the 
> archive.
>
> Now, B doesn't build while A builds.
>
> Whose fault is that? There is not enough information in a single build 
> to prove that, so we have to build again, but changing the constrains.
>
> If A builds but B doesn't, there are two possibilies:
>
>  1) B broke itself
>  2) A broke B
>
> Since we know that B-yesterda built against A-yesterday, than the 
> result of the B-today against A-yesterday will tell us if it's 1) or 2)
>
> Ok, with two projects is easy, what about three?
>
>  C -> B -> A
>
> worked yesterday, but today C is broken and B and A work. Who broke 
> the build?
>
> First of all, is it safe to assume that since A is shielded by B and 
> both B and A work, A has no influence on C?
>
> If so, we just apply the same algorithm for [C -> B].
>
> thoughts?
>
One step at a time,
Lets just getting gump using the last built jar,  and see what kind of 
patterns emerge.
R,
Nick