FW: [REPORT] Issue Clearance Quality + Technical Debt

["Dennis E. Hamilton" <or...@apache.org>]

I was forgetful.

The data summary is available in a PDF at <http://s.apache.org/YFT>.  A qualitative analysis is at <http://s.apache.org/SNg>.  The following note is more comprehensive than that.

It is helpful, if you please, to bring discussion of this topic to the dev@ list so the discussion is all in one place.

 - - - - - - - - - - - - - - - - - - - - - Forwarded Note - - - - - - -

To: dev@openoffice.apache.org
[BCC to Project Management Committee and users@ oo.a.o]

SUMMARY

The top-level analysis of Bugzilla issue handling has been completed for all issues opened on the project through December 31, 2015.  

The complete tabulation is in the PDF document at <http://s.apache.org/YFT>.

It remains the case that since the establishment of Apache OpenOffice as an ASF Top Level Project in November, 2012, the accrual of unresolved issues exceeds 40%.  That is, for every 100 new issues, on the average more than 40 of them will be unresolved indefinitely.  

In contrast, although there is a very large number of unresolved issues that remain in the Bugzilla from its history as part of OpenOffice.org, that previous technical debt was, proportionally, under 20%.  

Some highlights:

 * Even though the monthly rate of new issues and comments on issues has been decreasing significantly since mid-2014, the rate of technical debt as the proportion of unresolved issues has not improved.  

 * Although a reduction to 35% unresolved-issue is seen in the last 5 months of 2015, this may be distorted by issues created and then resolved in the staging of AOO 4.1.2 release candidates and QA on the candidates.  Results for the first-quarter of 2016 are needed to determine if this is a new trend or a hiccup.

DETAIL AND QUALITY MATTERS

This is a rough analysis, although the consistent trend is difficult to explain away.  

Refinement requires a closer look at the nature of issues and understanding of exactly what resolution means, not just what being left unresolved means.  

There is also a suspected disconnect with regard to what is considered an issue and how the ways of closing an issue are actually applied.  

 * Closing of a new issue as a duplicate qualifies as a resolution.  The incidence of long-standing issues that continue to receive duplicate reports is useful to understand in this case, and that requires more detail.

 * Some issues are closed as Resolved Fixed when the fact of the matter is that there was insufficient detail to understand and confirm the issue and the reporting party failed to provide additional information (if it was even requested).

 * Some comments on issues tailgate possibly-different problems onto known ones, although the resemblance may be superficial and the issues need to be split.

 * Enhancement/feature requests are not distinguished.

 * Resolved issues are sometimes closed without obtaining confirmation that incorporation of the identified resolution in distributed code actually addresses the originally-reported difficulty.

 * Some issue reports are closed as not issues because they are declared user problems and kicked to the Community Forum.

These may all have small effects.  We will know only by looking more closely into Bugzilla details.

The last case deserves more careful attention.  

The next-in-line users of Apache OpenOffice distributions consist of around 50 million users who are mainly individuals and 87% of whom are using Microsoft Windows.  Such casual users, whatever their limited experience in trouble-shooting and describing problems, are the main users of this software.  The usability issues they encounter are important to the project; even though they may not involve bugs in the code, they point to defects in the product.  Capturing those experiences and recognizing them as real issues for the user community is important.  That feedback is important for determining and making available workarounds and advice.  It can also inform changes to the software that might ameliorate those difficulties for non-experts.  Here's a simple example: having an easy-to-use option for resetting the user profile.

FUTURE STEPS

Along with extending the current analysis into the first quarter of 2016, exploration of the nature of unresolved and resolved issues will be introduced.


 - Dennis




> -----Original Message-----
> From: Dennis E. Hamilton [mailto:orcmid@apache.org]
> Sent: Wednesday, September 9, 2015 11:33
> To: dev@openoffice.apache.org
> Subject: FW: [DISCUSS] SUSTAINABILITY: Issue Clearance Quality +
> Technical Debt
> 
> 
> From the Chair,
> 
> From "A Maturity Model for Apache Projects,"
> <http://community.apache.org/apache-way/apache-project-maturity-
> model.html>
> 
>     "QU50: The project strives to respond to documented bug reports
>            in a timely manner."
> 
> It is also a recommended practice to acknowledge receipt of reports
> within some minimum time period, sometimes 24 hours, certainly not more
> than 72 hours.
> 
> This note discusses crude estimation of the technical debt that the
> project is accruing in terms of unresolved issues in the Bugzilla
> system.  It is important to appreciate the statistics as warning
> indicators.  It is necessary to look deeper for better understanding.
> 
> The risk of not attending to these indications is exposure of users to a
> growing set of known defects and related issues that remain in the
> software indefinitely.
> 
> DISCUSSION
> 
> Besides the questions posed by growing technical debt, below, perhaps
> the most pressing topic for resolution this year is Improved
> Coordination of Issues, near the end of this lengthy note.  This
> requires a community approach and a group of contributors able to work
> with users in providing first-line support in identification of issues,
> recording issues, and directing users directly to known solutions or
> work-arounds.  The volunteer users who accomplish that will require all
> of our support.  They are also our best source of on-the-ground
> experience with regard to what the major difficulties are.
> 
> All questions, proposals, corrections, and suggestions are welcome.
> 
> TECHNICAL DEBT
> 
> Using crude statistics from an earlier report, every indication is that
> the rate of unresolved issues is continuing to accrue as technical debt
> with over 40% of arriving issues unresolved.  In this case, the debt is
> the accumulation of unresolved defects and the cost of a process that
> reduces the increase while also paying down past debt.
> 
> As of 2015-08-05
>   * the oldest unresolved issue is #497 created
>     2001-03-02
>   * 24115/121298 (20%) issues unresolved from before
>           November, 2012
>   * 2232/5139 (43%) issues unresolved of
>           new issues from November, 2012 (after TLP)
>           through July, 2015
>   * 192/452 (42%) issues unresolved
>           of those created in the first
>           7 months of 2015
> 
> There are a number of factors that impact the quality of these
> statistics, including
> 
>   * Unresolved issues are not differentiated by type.
>     Feature and enhancement requests are counted among
>     confirmed defects.
> 
>   * The resolved portion includes issues rejected as
>     duplicates, as lacking supporting information,
>     and as usability/support matters rejected as not
>     being bugs, even when obviously about defects.
> 
> There is every reason to fear that, after such adjustments, the rate of
> technical debt accrual is even greater than 42-43%.
> 
> CONTEXT
> 
>   2012-11: #121299 First new issue in the Bugzilla of the AOO Top Level
> Project.
>   2015-07: #126439 Last new issue in the Bugzilla at the end of July,
> 2015.
> 
> By years at the TLP, (2012 and 2015 partial)
> 
>  2012    2013   2014   2015
> 
>   929    2136   1739    441 BZ items/month
>   133     198    170     65 New issues/month
> (averages are rounded to whole numbers)
> 
> The rate of arrival of new issues is steadily declining from the 2013
> peak.  That is not necessarily good news.  It does not demonstrate
> better proportional clearing of the smaller number.
> 
> PROPOSED MITIGATION
> 
> The project is currently in a crunch to release Apache OpenOffice 4.1.2.
> There will be a bump in Bugzilla activity after which there can be a
> fresh look at this situation and the developers now heads-down on 4.1.2
> can look to the horizon beyond.
> 
> MONITORING.  These data will be updated monthly, using the July 2015
> state as a baseline with respect to changes in the technical debt and
> rate of increase.  A new baseline from December 2015 can be used for
> 2016, continuing on an annual basis thereafter.
> 
> REVIEW.  Some drill-down is required to differentiate between unresolved
> feature and enhancement requests in contrast to reports that relate to
> defects of some kind.  It is also important to understand the incidence
> of duplicates for still-unresolved issues and issues that are marked
> resolved without any action having taken place.
> 
> IMPROVED COORDINATION OF ISSUES.  Issues involve far more than
> identified bugs.  Reported issues are valuable information about
> experiences of users and others and where their difficulties arise.
> There is no means to determine how many others experience an issue and
> do not report it.
>    It is important that usability issues of many kinds be captured,
> tracked and mitigated wherever possible.  Appropriate categories should
> be added to the Bugzilla and issues of such kinds should be embraced as
> the gifts that they are.  Simple additions to the current Bugzilla
> classifications will be investigated.
>    In addition, many issues are recognized on the mailing lists and
> forums.  It is important that there be cross-talk among the different
> volunteer supporters and that there be no direction of users to other
> places instead of providing some positive assistance.  This involves
> more engagement and communication among those who work with the
> Bugzilla, the mailing lists, and the forums to ensure that all avenues
> are covered.  It is desirable for users to be supported with positive
> assistance no matter how they bring their situation to the attention of
> the project.  Accomplishing that is open for discussion.  "It takes a
> village."
> 
> 
> -----Original Message-----
> From: Dennis E. Hamilton [mailto:orcmid@apache.org]
> Sent: Wednesday, August 5, 2015 19:57
> To: dev@openoffice.apache.org
> Subject: STATE OF AOO: Overall Bugzilla Activity through July 2015
> 
> In looking for visible indicators of project activity, I created an
> overview of Bugzilla activity from November 2012 through July 2015.
> 
> This is a high-level view of gross activity and does not provide fine
> details.  There is still an interesting picture.
> 
> My complete tabulation is available in a PDF document at
> <http://svn.apache.org/repos/asf/openoffice/pmc/project-state/2015-07-
> BZ-OverallActivity-2015-08-05-dh.pdf>.
> 
> [ ... ]
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
For additional commands, e-mail: qa-help@openoffice.apache.org