You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-issues@jackrabbit.apache.org by "Alex Parvulescu (JIRA)" <ji...@apache.org> on 2012/10/24 18:38:13 UTC
[jira] [Created] (OAK-394) IndexManagerHook to manage existing
indexes
Alex Parvulescu created OAK-394:
-----------------------------------
Summary: IndexManagerHook to manage existing indexes
Key: OAK-394
URL: https://issues.apache.org/jira/browse/OAK-394
Project: Jackrabbit Oak
Issue Type: Sub-task
Reporter: Alex Parvulescu
Assignee: Alex Parvulescu
The Index Manager is making a comeback!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-394) IndexManagerHook to manage existing
indexes
Posted by "Alex Parvulescu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490560#comment-13490560 ]
Alex Parvulescu commented on OAK-394:
-------------------------------------
A tiny update: I'm not happy with the way this works now, I think I missed some important bits tied to the way diffs work.
The IndexManager gathers the IndexDefinitions first, then it passes them via the IndexHookProvider to relevant IndexHook impls.
The problem here is that the NodeState that gets passed on is the index content node (ex: "/oak:index/index"), when in fact it should be the repository node that needs to be indexed (ex: "/").
Another problem is that recursing through the diffs doesn't work properly. Even if the IndexManager succeeds in gathering all the index defs under certain paths, the path info is never used, so triggering a full reindex can mean that "all" the index implementation (never mind the location) will contain all the repository content.
This is not a big problem now, as we only have a few index defs under the root.
> IndexManagerHook to manage existing indexes
> -------------------------------------------
>
> Key: OAK-394
> URL: https://issues.apache.org/jira/browse/OAK-394
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Reporter: Alex Parvulescu
> Assignee: Alex Parvulescu
> Attachments: indexmanager-commondiff.patch, indexmanager-commondiff-v2.patch, indexmanager.patch
>
>
> The Index Manager is making a comeback!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-394) IndexManagerHook to manage existing
indexes
Posted by "Alex Parvulescu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483982#comment-13483982 ]
Alex Parvulescu commented on OAK-394:
-------------------------------------
thanks for the review, Jukka!
I agree with your point, I will push the code as is to have the mechanism in place, then follow up with another issue regarding optimization of the IndexManager's diff mechanism.
The optim could also include the following ideas:
- IndexHooks won't really have to extend CommitHooks anymore, this way they could be made to work with diffs directly.
- have a common diff mechanism that allows for this kind of single-pass-update-all-indexes behaviour that is missing currently
> IndexManagerHook to manage existing indexes
> -------------------------------------------
>
> Key: OAK-394
> URL: https://issues.apache.org/jira/browse/OAK-394
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Reporter: Alex Parvulescu
> Assignee: Alex Parvulescu
> Attachments: indexmanager.patch
>
>
> The Index Manager is making a comeback!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-394) IndexManagerHook to manage existing
indexes
Posted by "Alex Parvulescu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487657#comment-13487657 ]
Alex Parvulescu commented on OAK-394:
-------------------------------------
pushed the common diff change with rev 1404071.
> IndexManagerHook to manage existing indexes
> -------------------------------------------
>
> Key: OAK-394
> URL: https://issues.apache.org/jira/browse/OAK-394
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Reporter: Alex Parvulescu
> Assignee: Alex Parvulescu
> Attachments: indexmanager-commondiff.patch, indexmanager-commondiff-v2.patch, indexmanager.patch
>
>
> The Index Manager is making a comeback!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-394) IndexManagerHook to manage existing
indexes
Posted by "Alex Parvulescu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483994#comment-13483994 ]
Alex Parvulescu commented on OAK-394:
-------------------------------------
first draft in with revision 1402036.
> IndexManagerHook to manage existing indexes
> -------------------------------------------
>
> Key: OAK-394
> URL: https://issues.apache.org/jira/browse/OAK-394
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Reporter: Alex Parvulescu
> Assignee: Alex Parvulescu
> Attachments: indexmanager.patch
>
>
> The Index Manager is making a comeback!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OAK-394) IndexManagerHook to manage existing
indexes
Posted by "Alex Parvulescu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Parvulescu updated OAK-394:
--------------------------------
Attachment: indexmanager.patch
proposed patch up for review.
More or less as discused with Jukka offline :)
> IndexManagerHook to manage existing indexes
> -------------------------------------------
>
> Key: OAK-394
> URL: https://issues.apache.org/jira/browse/OAK-394
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Reporter: Alex Parvulescu
> Assignee: Alex Parvulescu
> Attachments: indexmanager.patch
>
>
> The Index Manager is making a comeback!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-394) IndexManagerHook to manage existing
indexes
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483985#comment-13483985 ]
Jukka Zitting commented on OAK-394:
-----------------------------------
bq. follow up with another issue
I'd just leave this issue open until the code stabilizes.
> IndexManagerHook to manage existing indexes
> -------------------------------------------
>
> Key: OAK-394
> URL: https://issues.apache.org/jira/browse/OAK-394
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Reporter: Alex Parvulescu
> Assignee: Alex Parvulescu
> Attachments: indexmanager.patch
>
>
> The Index Manager is making a comeback!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (OAK-394) IndexManagerHook to manage existing
indexes
Posted by "Alex Parvulescu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Parvulescu resolved OAK-394.
---------------------------------
Resolution: Fixed
Fix Version/s: 0.6
Not much I can add to this, if there are other aspects to consider, I'll file separate issues.
> IndexManagerHook to manage existing indexes
> -------------------------------------------
>
> Key: OAK-394
> URL: https://issues.apache.org/jira/browse/OAK-394
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Reporter: Alex Parvulescu
> Assignee: Alex Parvulescu
> Fix For: 0.6
>
> Attachments: indexmanager-commondiff.patch, indexmanager-commondiff-v2.patch, indexmanager.patch
>
>
> The Index Manager is making a comeback!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OAK-394) IndexManagerHook to manage existing
indexes
Posted by "Alex Parvulescu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Parvulescu updated OAK-394:
--------------------------------
Attachment: indexmanager-commondiff-v2.patch
common diff mechanism v2
> IndexManagerHook to manage existing indexes
> -------------------------------------------
>
> Key: OAK-394
> URL: https://issues.apache.org/jira/browse/OAK-394
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Reporter: Alex Parvulescu
> Assignee: Alex Parvulescu
> Attachments: indexmanager-commondiff.patch, indexmanager-commondiff-v2.patch, indexmanager.patch
>
>
> The Index Manager is making a comeback!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-394) IndexManagerHook to manage existing
indexes
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483977#comment-13483977 ]
Jukka Zitting commented on OAK-394:
-----------------------------------
+1 looks good, though I'd follow up with the following change:
Currently the IndexHookManager causes a lot of traversals of the entire change set. First with the {{IndexDefDiff}} class to find all indexes that need updating and then once for all of those indexes. That's potentially a lot of {{compareAgainstBaseState()}} calls that could get a bit expensive especially for large transactions. With some further refactoring we should change the code so that the IndexHookManager traverses the change set just once, collecting affected indexes and dispatching updated notifications to them all in just one go.
> IndexManagerHook to manage existing indexes
> -------------------------------------------
>
> Key: OAK-394
> URL: https://issues.apache.org/jira/browse/OAK-394
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Reporter: Alex Parvulescu
> Assignee: Alex Parvulescu
> Attachments: indexmanager.patch
>
>
> The Index Manager is making a comeback!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-394) IndexManagerHook to manage existing
indexes
Posted by "Alex Parvulescu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493938#comment-13493938 ]
Alex Parvulescu commented on OAK-394:
-------------------------------------
I finally found a working solution that replies on a single diff, which in turn pushes the changes to the existing index implementations.
The change is rather involved, if I missed anything please let me know.
The integration tests pass, change is in with rev 1407425.
Parts of the code are very similar, I suppose some methods could be extracted into a sort of util class.
> IndexManagerHook to manage existing indexes
> -------------------------------------------
>
> Key: OAK-394
> URL: https://issues.apache.org/jira/browse/OAK-394
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Reporter: Alex Parvulescu
> Assignee: Alex Parvulescu
> Attachments: indexmanager-commondiff.patch, indexmanager-commondiff-v2.patch, indexmanager.patch
>
>
> The Index Manager is making a comeback!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OAK-394) IndexManagerHook to manage existing
indexes
Posted by "Alex Parvulescu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484027#comment-13484027 ]
Alex Parvulescu commented on OAK-394:
-------------------------------------
osgi annotations + javadocs with revision 1402088
> IndexManagerHook to manage existing indexes
> -------------------------------------------
>
> Key: OAK-394
> URL: https://issues.apache.org/jira/browse/OAK-394
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Reporter: Alex Parvulescu
> Assignee: Alex Parvulescu
> Attachments: indexmanager.patch
>
>
> The Index Manager is making a comeback!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OAK-394) IndexManagerHook to manage existing
indexes
Posted by "Alex Parvulescu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OAK-394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Parvulescu updated OAK-394:
--------------------------------
Attachment: indexmanager-commondiff.patch
patch to address the common diff problem.
I'm managed to get the diff process down to 2 (possible 3 diffs)
first one is the index definition diff, second one is a composite diff that delegates all events to the available the index implementations at the same time.
the 3rd diff comes in when we are dealing with a reindex.
The IndexHook#postProcess(NodeState) only takes a param because of the lucene impl - the param will probably go away soon.
> IndexManagerHook to manage existing indexes
> -------------------------------------------
>
> Key: OAK-394
> URL: https://issues.apache.org/jira/browse/OAK-394
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Reporter: Alex Parvulescu
> Assignee: Alex Parvulescu
> Attachments: indexmanager-commondiff.patch, indexmanager.patch
>
>
> The Index Manager is making a comeback!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira