[jira] Updated: (GUMP-21) FOG factor is misleading and requires improvement

["Leo Simons (JIRA)" <ge...@gump.apache.org>]

     [ http://issues.apache.org/jira/browse/GUMP-21?page=history ]

Leo Simons updated GUMP-21:
---------------------------

    Version: unspecified

> FOG factor is misleading and requires improvement
> -------------------------------------------------
>
>          Key: GUMP-21
>          URL: http://issues.apache.org/jira/browse/GUMP-21
>      Project: Gump
>         Type: Improvement
>   Components: Python-based Gump
>     Versions: unspecified
>     Reporter: Daniel Savarese
>     Assignee: Adam Jack

>
> FOG is defined right now as:
> def getFOGFactor(self):
>     return round((float(self.successes) / (float(self.failures) + float(self.prereqs))), 2)
> This is not helpful.  Last month, jakart-oro
> failed to build by coincidence around the time of a Gump
> run and the problem was fixed before the next Gump run,
> yet its FOG factor dropped in half.  The responsiveness
> to resolving build and integration problems is not taken
> into account.  It is likely that it is impossible for FOG
> to mean anything because the factors that determine
> a project's "health" (to borrow the term used by Adam Jack
> in an email to the commons-dev and gump lists) are largely
> orthogonal.  FOG should probably be a vector and if you really
> wanted a single number you could use the length of that vector.
> One of the major problems with the current factor is that
> a project's FOG factor continues to increase when no changes
> have been made to the repository.  It's not meaningful to
> count only successful builds towards improving the factor
> because no changes may have been made to the code base.
> Ideally, you count only builds that cause no failures after
> a change to the code base.  FOG doesn't factor in how quickly
> (or slowly) projects recover from build failures, and how often
> changes to a code base cause failures (either in itself or in
> other projects).  Right now, a project with few code changes
> (say once a month) that causes a failure 50% (1 every 2 months)
> of the time the code is changed with a mean time of failure
> correction of 1 week is deemed more reliable (higher FOG factor)
> than a project that makes frequent changes (say every day) to
> its code base and causes a failure 10% of the time (1 every 10
> days) with a mean time of failure correction of 1 day.  That's
> because FOG is based almost entirely on consecutive successful
> Gump builds, which is not useful.
> Rather than propose a new meaningless way of computing FOG, I
> suggest FOG stop being used temporarily.  It would be more
> useful to break out and report all of the independently useful
> metrics and through experience over time determine how each
> impacts the overall notion of FOG and eventually derive a
> way of condensing the information (perhaps a FOG factor,
> perhaps a vector) in a useful manner.  Right now the state
> of affairs is that someone decided FOG was a cool thing, so
> it's being reported and people see this number and mistakenly
> think a project with half the FOG factor of another is
> less dependable and therefore avoid using it.  Alternatively
> an explanation of what FOG is intended to compute and how
> it is computed should be placed on Gump reports that list
> FOG so that consumers of the information will not be misled.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org