[jira] [Updated] (OAK-1970) Optimize the diff logic for large number of children case

["Chetan Mehrotra (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OAK-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chetan Mehrotra updated OAK-1970:
---------------------------------
    Fix Version/s:     (was: 1.2)
                   1.3.0

> Optimize the diff logic for large number of children case 
> ----------------------------------------------------------
>
>                 Key: OAK-1970
>                 URL: https://issues.apache.org/jira/browse/OAK-1970
>             Project: Jackrabbit Oak
>          Issue Type: Improvement
>          Components: mongomk
>            Reporter: Chetan Mehrotra
>            Assignee: Chetan Mehrotra
>             Fix For: 1.3.0
>
>
> DocumentNodeStore currently makes use of query to determine child nodes which have changed after certain time. Query used is something like
> {noformat}
> db.nodes.find({ _id: { $gt: "3:/content/foo/01/", $lt: "3:/content/foo010" }, _modified: { $gte: <start time> } }).sort({_id:1})
> {noformat}
> OAK-1966 tries to optimize the majority case where start times is recent and in that case it makes use of _modified index. However if the start time is quite old and a node has large number of children say 100k then it would involve scan of all those 100k nodes as _modified index would not be of much help. 
> Instead of querying like this we can have a special handling for cases where large number of children are involved. It would involve following steps
> After analyzing the runtime queries in most case it is seen that even with old modified time the number of change nodes is < 50
> # Mark parent nodes which have large number of children say > 50
> # On such nodes we would keep an array of \{modifiedtime, childName\} ## Array would be bounded say keep last 50 updates. This can be done via splice and push operators [1]
> ## Each entry in array would record modifiedtime and name of child node which was modified. 
> ## Array would be sorted on modifiedtime
> # Each updated to any child belonging to such parent would also involve update to above array
> # When we query for modified we check if the parent has such an array (if parent is in cache) and if that array has time entries from the required start time we directly make use of that and avoid the query
> This should reduce needs for such queries in majority of cases
> [1] http://docs.mongodb.org/manual/reference/operator/update-array/
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)