Automatic index cardinality statistics update - changes for 10.8

[Kristian Waagan <kr...@oracle.com>]

Hi,

The 10.8 release candidate will soon be created, and there are a few 
remaining decisions to make regarding the new "istat feature" (automatic 
update/creation of index cardinality statistics):
   a) Documentation, see DERBY-5083 and comments under DERBY-5051.
   b) Whether or not to enable logging by default, especially the 
processing stats summary printed on database shutdown. See comment under 
DERBY-5051.
   c) Whether or not to increase the table row count thresholds for 
statistics creation and update. These are currently 100 and 1000, 
respectively.


I don't have many cycles to spend before the RC is cut, but here are my 
thoughts:

   a) Now that a release note has been written, we probably have the 
minimum amount of documentation required. We probably don't want too 
much documentation in the manual(s), since the functionality may change 
based on user feedback from 10.8. As long as the user is told how to 
disable the feature, I think we're good. If someone wants to describe 
more of the feature, things like logging, tracing, and how 
updates/creations are triggered may be good points to cover.

   b) I see that Knut argued logging should be disabled by default in 
the production code. That's fine with me, the only thing we may loose 
[1] is basic information in the case of intermittent and rare scenarios.

   c) Since the thresholds are set based on gut-feeling, we may want to 
err on the side of caution instead of burning resources on generating 
statistics without much gain. Assuming the update threshold (basically 
the minimum number of rows in the table) is kept reasonably low, I would 
set the creation threshold  to the same value. The optimal value would 
be when having more accurate statistics would make the optimizer choose 
a significantly better plan.  An always correct value may not exist... 
Any suggestions?

Besides from these issues, I observe that one of the istat tests runs 
into an OOME on Java 5.0 and 1.4 (DERBY-5057), one of the istat tests 
fails on Windows due to problems deleting files [2], and one other 
intermittent test failure (DERBY-5045).
Did I miss anything?


Regards,
-- 
Kristian

[1] Assuming the user keeps the logs and comes to the Derby community 
for help.
[2] Hopefully masked/fixed by DERBY-4915 (will commit very soon).