User community role 10.2 testing of optimizer changes

[Kathey Marsden <km...@sbcglobal.net>]

In the licencing discussion,  I mentioned that we really need more user 
feedback before we release 10.2.
Discussion has occurred on the developer list  on that point and is on 
this thread. 
http://www.nabble.com/10.2-plans-%28was-Re%3A-10.2-licensing-issue%29-tf2256208.html

The summary is that right now the development community is not in a 
position to work on any known regression and we should release to 
motivate users to try 10.2. Kathy Saunders probably summed it up the 
concern best when she said:

"If we really believe we need more testing, then what is that testing 
and who is going to do it? "

My opinion is that our overall confidence in the  quality  and 
robustness of Derby needs to increase release to release and we should 
feel confident about that and should communicate risk in areas that are 
likely impacted in terms that users can understand so you can either  
adjust your  expectations or do your part to bring your expectations 
back to their prior level.  Whether you count on Derby to control your 
I.V drip, use it in the critical path of your business,  or just use it 
to manage your CD collection, you have the right to know the information 
and can then assess whether you want to take your necessary role of 
flushing out optimizer issues, before or after the release.     The 
summary is this:

There were significant optimizer performance changes in 10.2 that are 
taking queries that were running for hours to seconds. (Maybe someone 
can point to the data).  These changes involve the optimizer and it is 
really not possible to have comprehensive regression tests in this 
area.  The changes in my opinion are  worth while, high quality, risky 
and regression prone.    We have had optimizer  feedback from a single 
user who exposed several issues. Other issues have been exposed by 
development.  We have fixed what we can.  We need users  especially 
those who have complex or performance sensitive queries to try their 
existing applications and give us feedback.  Army can you please give an 
overview of the changes from a functional perspective and explain what 
types of usage you think could pop issues?  Also I would like your 
opinion on the value of such testing from the user community?

Users, please register your results at:
http://wiki.apache.org/db-derby/TenTwoApplicationTesting
 
Kathey

Re: User community role 10.2 testing of optimizer changes

[Kathey Marsden <km...@sbcglobal.net>]

Kathey Marsden wrote:

>   We need users  especially those who have complex or performance 
> sensitive queries to try their existing applications and give us 
> feedback.

Yikes,  I forgot the beta link.
http://people.apache.org/~rhillegas/10.2.1.3-beta/

Other 10.2 information
http://wiki.apache.org/db-derby/TenTwoRelease
http://wiki.apache.org/db-derby/TenTwoSnapshot

Register your application testing results and the type of derby usage 
you work think they work out at
http://wiki.apache.org/db-derby/TenTwoApplicationTesting

File issues and list them for the contest at
http://wiki.apache.org/db-derby/RegressionSearchAndDestroy

If we can get  testing of twenty  query intensive applications  I would 
feel better about a release.  I see  the issues  that  have come in 
recently
as a indicator  that we need a larger data set to understand the 
likelihood that users will hit regressions when they upgrade.  Please help.

Thanks

Kathey

Re: User community role 10.2 testing of optimizer changes

["Jean T. Anderson" <jt...@bristowhill.com>]

Kathey Marsden wrote:
> In the licencing discussion,  I mentioned that we really need more user
> feedback before we release 10.2.
...
> Users, please register your results at:
> http://wiki.apache.org/db-derby/TenTwoApplicationTesting

or, alternatively, how about posting feedback to derby-user?

Anyone can create a Wiki profile at http://wiki.apache.org/db-derby .
But I can appreciate that those who don't already know MoinMoin wiki
syntax (or similar syntax from other wiki software) might decide to add
feedback "later" when they can find time to spin up on it. And we all
know how soon "later" sometimes happens.  :-)

 -jean

Re: User community role 10.2 testing of optimizer changes

[Kathey Marsden <km...@sbcglobal.net>]

Army wrote:

>
> Users may also want to remember the policy of "scratch your own itch" 
> or "fry your own fish": I as a developer tend to have more time and 
> inclination to address issues with contributed code closer to the time 
> I actually made the contribution.  

This is a good point and I bet your immediate  understanding and ability 
to turn around fixes is  better when you are actively working on it.
It is really critical that users test the beta now and not later but 
already it is very late.  An even earlier partnership between the user 
community and development would have greatly reduced the regression risk 
for this change.  Folks  who wanted this improvement could have signed 
up to test along the way and we could have gotten much more input.  We 
treat our user/developer relationship  too much like a fine dining 
establishment.  Developers  deliver a dish cooked,  users  take a bite 
and see if they like it and then send it back if  they don't.  We need 
our users in the kitchen tasting  as we go along.  Army is right. If the 
dish is free and the cook was cooking for fun,  it is hard to get a 
rebake.  Please test now.

http://people.apache.org/~rhillegas/10.2.1.3-beta/

For other popular performance and feature improvements like  DERBY-47,  
I hope we can use  a different  model.  A development/user partnership 
where users get the high quality improvements they want by providing 
development the testing support needed  along the way and development 
communicates where user help is needed and most valuable and where they 
are at risk.   We break down the wall between the kitchen and the dining 
room and work together to product "consistently high quality software."

For now we need users  urgent help to try 10.2 beta.  It's 5:55 and 
dinner is scheduled at 6:00.  We have a feeling  something is wrong but 
don't know what it is or what to do about it.   Please come take a taste 
before we put it on the table.

Thanks

Kathey

Re: User community role 10.2 testing of optimizer changes

[Army <qo...@gmail.com>]

Kathey Marsden wrote:
> We need users especially those who have complex or performance 
> sensitive queries to try their existing applications and give us 
> feedback.  Army can you please give an overview of the changes 
> from a functional perspective and explain what types of usage 
> you think could pop issues?

For "an overview of the changes from a functional perspective", users should see 
the problem descriptions attached to the relevant Jira issues:

https://issues.apache.org/jira/browse/DERBY-805
   --> DERBY-805_v5.html, sections I and II

https://issues.apache.org/jira/browse/DERBY-781
   --> DERBY-781_v1.html, sections I and II

As for the "types of usage", there are generally three areas that are most 
directly affected by the optimizer changes:

1. Queries with UNIONs in them (the more unions, the more likely the query is to 
be affected).

2. Queries with subqueries in them, either explicitly or indirectly through 
views.  The more deeply nested subqueries there are, the more likely the query 
is to be affected.

3. Queries which perform joins in which at least one of the result sets to be 
joined is a UNION, a subquery, or a combination of the two.

Thus far all optimizer regressions have been discovered by queries posted by a 
single user whose app uses large, deeply-nested queries that involve all three 
of the above areas.  See in particular DERBY-1205, DERBY-1633, DERBY-1777.

That said, it's possible that other queries will also be affected since 
additional optimizer-related bug fixes have been made--esp. DERBY-1007 and 
DERBY-1357.  So as Kathey said, anyone with query-intensive applications might 
find it beneficial to do some testing with 10.2.

> Also I would like your opinion on the value of such testing 
> from the user community?

Very valuable, no doubt there.  If users don't test it beforehand, they run the 
risk of finding problems the hard way.  I know it can take a lot of time and 
effort to test a beta candidate, so it's not too surprising that we've had 
little response to Derby's multiple requests for more user testing.  I guess 
it's up to the user to decide when and how the time and effort is going to be 
spent: early on in beta, or later when regressions are (presumably) more critical.

Users may also want to remember the policy of "scratch your own itch" or "fry 
your own fish": I as a developer tend to have more time and inclination to 
address issues with contributed code closer to the time I actually made the 
contribution.  If the code is untested for several months and then a user hits a 
regression at release time, what am I going to be doing at that time?  And am I 
going to have the time and means to resolve the problem right then?  Will that 
regression be my "itch" or my "fish"?  I would of course hope the answer is an 
immediate "Yes"--but there's no guarantee in the world of opensource, and a lot 
can happen in a couple of months.

So yes, more testing is better.  Thanks to Kathey for continuing to push for 
more user feedback.  It can only help.

Army