[GitHub] [couchdb] davisp edited a comment on pull request #3164: Feature ebtree views

[GitBox <gi...@apache.org>]

davisp edited a comment on pull request #3164:
URL: https://github.com/apache/couchdb/pull/3164#issuecomment-700982451


   @garrensmith A bit of profiling showed that I was doing a bit of extra work compared to #3018 so there were a few simple optimizations to be made. Namely:
   
   1. Avoid calculating unused reductions while reading from a reduce view (coincidentally an optimization that exists even in the legacy view engine)
   2. Avoid some extra work in couch_query_servers that I re-used instead of duplicating the functionality of this PR.
   
   For 1, the #3018 PR achieves this same end by having a separate ebtree instance for each reduce function on the view. For 2, its similar other than the size calculation is done on at write time.
   
   ![Screen Shot 2020-09-29 at 3 18 02 PM](https://user-images.githubusercontent.com/19929/94612436-b7e49f80-0268-11eb-97e1-549b8b5ec80b.png)
   
   Updated numbers generated locally. I ported the Node.js script to Python [1] and removed deleting the database part. I then ran it four times. Once to generate the index, and then three times to gather numbers after everything was warmed up in cache to try and get an equal footing for each replica.
   
   There's still about 200ms difference between the two implementations. However, based on profiling both implementations it would appear that a good chunk of that is due to #3018 performing significantly fewer invocations of collation which is a bug in a previous version of ebtree that has since been fixed [2]. I also went fixed the lack of JSON collation in #3018 [3] and that appears to have made it faster.
   
   I'm fairly confident I've covered the differences given the similar algorithmic properties of the implementations. Let me know if you're OK merging this now.
   
   [1] https://gist.github.com/davisp/48303dcd877b31632fb75055d6758129
   [2] https://github.com/apache/couchdb/commit/f8fdf9721e2ac932022065bc075301641568d67c
   [3] https://github.com/cloudant/couchdb/compare/fdb-btree-reduce...apache:fdb-btree-reduce-davisp


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org