You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org> on 2009/08/04 15:11:14 UTC
[jira] Created: (SOLR-1331) Support merging multiple cores
Support merging multiple cores
------------------------------
Key: SOLR-1331
URL: https://issues.apache.org/jira/browse/SOLR-1331
Project: Solr
Issue Type: New Feature
Components: update
Reporter: Shalin Shekhar Mangar
Fix For: 1.5
There should be a provision to merge one core with another. It should be possible to create a core, add documents to it and then just merge it into the main core which is serving requests. This way, the user will not need to know the filesystem as it is needed for SOLR-1051
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1331) Support merging multiple cores
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742581#action_12742581 ]
Hoss Man commented on SOLR-1331:
--------------------------------
bq. I was thinking about more safe-guards like acquiring locks on source cores to make sure a writer is not opened on them until the merge completes.
Ah... gotcha. yeah, it makes sense that the existing merge API (that requires you to have an index path) can get away with not worrying about this (making it the callers responsibility) but a more user-freindly api for merging by core name should probably hide this from the user.
another option might just be to require that the (src) core is UNLOADed first.
> Support merging multiple cores
> ------------------------------
>
> Key: SOLR-1331
> URL: https://issues.apache.org/jira/browse/SOLR-1331
> Project: Solr
> Issue Type: New Feature
> Components: update
> Reporter: Shalin Shekhar Mangar
> Fix For: 1.5
>
>
> There should be a provision to merge one core with another. It should be possible to create a core, add documents to it and then just merge it into the main core which is serving requests. This way, the user will not need to know the filesystem as it is needed for SOLR-1051
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1331) Support merging multiple cores
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742178#action_12742178 ]
Hoss Man commented on SOLR-1331:
--------------------------------
Is the thinking here that we just expose SOLR-1051 type functionality at the core level w/o using file paths -- in which case the caller is responsible for knowing whether or not the index merging makes logical sense; or are we talking about a more user friendly concept of core merging where we sanity check that the schemas are the same -- or even more complex: merge the schemas if they don't have any conflicts (ie: all field/type names in common have identical declarations?)
the first seems easy and essentially syntactic sugar on top of what was already implemented in SOLR-1051 (get core by name; get index dir from core; merge) but since Shalin opened a new issue for this, i suspect he's got something bigger in mind.
> Support merging multiple cores
> ------------------------------
>
> Key: SOLR-1331
> URL: https://issues.apache.org/jira/browse/SOLR-1331
> Project: Solr
> Issue Type: New Feature
> Components: update
> Reporter: Shalin Shekhar Mangar
> Fix For: 1.5
>
>
> There should be a provision to merge one core with another. It should be possible to create a core, add documents to it and then just merge it into the main core which is serving requests. This way, the user will not need to know the filesystem as it is needed for SOLR-1051
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1331) Support merging multiple cores
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742330#action_12742330 ]
Yonik Seeley commented on SOLR-1331:
------------------------------------
bq. However, I was thinking about more safe-guards like acquiring locks on source cores to make sure a writer is not opened on them until the merge completes.
IndexWriter.addIndexes() doesn't require this does it?
> Support merging multiple cores
> ------------------------------
>
> Key: SOLR-1331
> URL: https://issues.apache.org/jira/browse/SOLR-1331
> Project: Solr
> Issue Type: New Feature
> Components: update
> Reporter: Shalin Shekhar Mangar
> Fix For: 1.5
>
>
> There should be a provision to merge one core with another. It should be possible to create a core, add documents to it and then just merge it into the main core which is serving requests. This way, the user will not need to know the filesystem as it is needed for SOLR-1051
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1331) Support merging multiple cores
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742355#action_12742355 ]
Shalin Shekhar Mangar commented on SOLR-1331:
---------------------------------------------
bq. IndexWriter.addIndexes() doesn't require this does it?
It does. The javadocs for IndexWriter.addIndexesNoOptimize say:
{quote}
The index in each Directory must not be changed (opened by a writer) while this method is running. This method does not acquire a write lock in each input Directory, so it is up to the caller to enforce this.
{quote}
We can choose to leave this upto the user of the API or we can try to prevent it ourselves.
> Support merging multiple cores
> ------------------------------
>
> Key: SOLR-1331
> URL: https://issues.apache.org/jira/browse/SOLR-1331
> Project: Solr
> Issue Type: New Feature
> Components: update
> Reporter: Shalin Shekhar Mangar
> Fix For: 1.5
>
>
> There should be a provision to merge one core with another. It should be possible to create a core, add documents to it and then just merge it into the main core which is serving requests. This way, the user will not need to know the filesystem as it is needed for SOLR-1051
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1331) Support merging multiple cores
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742328#action_12742328 ]
Shalin Shekhar Mangar commented on SOLR-1331:
---------------------------------------------
bq. Is the thinking here that we just expose SOLR-1051 type functionality at the core level w/o using file paths - in which case the caller is responsible for knowing whether or not the index merging makes logical sense
Yes, that is the motivation. However, I was thinking about more safe-guards like acquiring locks on source cores to make sure a writer is not opened on them until the merge completes.
bq. are we talking about a more user friendly concept of core merging where we sanity check that the schemas are the same - or even more complex: merge the schemas if they don't have any conflicts (ie: all field/type names in common have identical declarations?)
That'd be cool but also more complex. We can do that if users ask for it.
> Support merging multiple cores
> ------------------------------
>
> Key: SOLR-1331
> URL: https://issues.apache.org/jira/browse/SOLR-1331
> Project: Solr
> Issue Type: New Feature
> Components: update
> Reporter: Shalin Shekhar Mangar
> Fix For: 1.5
>
>
> There should be a provision to merge one core with another. It should be possible to create a core, add documents to it and then just merge it into the main core which is serving requests. This way, the user will not need to know the filesystem as it is needed for SOLR-1051
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.