You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Mark Miller (JIRA)" <ji...@apache.org> on 2012/10/02 23:33:07 UTC
[jira] [Created] (SOLR-3911) Replication should work with any
Directory impl, not just local filesystem based Directories.
Mark Miller created SOLR-3911:
---------------------------------
Summary: Replication should work with any Directory impl, not just local filesystem based Directories.
Key: SOLR-3911
URL: https://issues.apache.org/jira/browse/SOLR-3911
Project: Solr
Issue Type: Improvement
Reporter: Mark Miller
Assignee: Mark Miller
Fix For: 4.1, 5.0
I've wanted to do this for a long time - there is no reason replication should not support any directory impl. This will let us use the mockdir for replication tests rather than having to force an FSDir and lose all the extra test checks and simulations. This will improve our testing around replication a lot, and allow custom Directory impls to be used on multi node Solr.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3911) Make Directory and DirectoryFactory
first class so that the majority of Solr's features work with any custom
implementations.
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474841#comment-13474841 ]
Mark Miller commented on SOLR-3911:
-----------------------------------
I think I'm close to committing something. Patch is getting big enough that it will be really annoying to keep up to date - so I'd like to commit and start baking in trunk as soon as possible. I'll post my latest work tomorrow.
> Make Directory and DirectoryFactory first class so that the majority of Solr's features work with any custom implementations.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-3911
> URL: https://issues.apache.org/jira/browse/SOLR-3911
> Project: Solr
> Issue Type: Improvement
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-3911.patch, SOLR-3911.patch, SOLR-3911.patch
>
>
> The biggest issue is that many parts of Solr rely on a local file system based Directory implementation - most notably, replication. This should all be changed to use the Directory and DirectoryFactory abstractions.
> Other parts of the code that count on the local file system for making paths and getting file sizes should also be changed to use Directory and/or DirectoryFactory.
> Original title: Replication should work with any Directory impl, not just local filesystem based Directories.
> I've wanted to do this for a long time - there is no reason replication should not support any directory impl. This will let us use the mockdir for replication tests rather than having to force an FSDir and lose all the extra test checks and simulations. This will improve our testing around replication a lot, and allow custom Directory impls to be used on multi node Solr.
> Expanded scope - full first class support for DirectoryFactory and Directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3911) Make Directory and DirectoryFactory
first class so that the majority of Solr's features work with any custom
implementations.
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-3911:
------------------------------
Attachment: SOLR-3911.patch
This issue has been very satisfying. All tests passing. I still force an fs directory for 2 solrcloud tests due to the recovery issue mentioned above. We can probably fix that in another issue.
> Make Directory and DirectoryFactory first class so that the majority of Solr's features work with any custom implementations.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-3911
> URL: https://issues.apache.org/jira/browse/SOLR-3911
> Project: Solr
> Issue Type: Improvement
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-3911.patch, SOLR-3911.patch, SOLR-3911.patch
>
>
> The biggest issue is that many parts of Solr rely on a local file system based Directory implementation - most notably, replication. This should all be changed to use the Directory and DirectoryFactory abstractions.
> Other parts of the code that count on the local file system for making paths and getting file sizes should also be changed to use Directory and/or DirectoryFactory.
> Original title: Replication should work with any Directory impl, not just local filesystem based Directories.
> I've wanted to do this for a long time - there is no reason replication should not support any directory impl. This will let us use the mockdir for replication tests rather than having to force an FSDir and lose all the extra test checks and simulations. This will improve our testing around replication a lot, and allow custom Directory impls to be used on multi node Solr.
> Expanded scope - full first class support for DirectoryFactory and Directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3911) Make Directory and DirectoryFactory
first class so that the majority of Solr's features work with any custom
implementations.
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-3911:
------------------------------
Description:
The biggest issue is that many parts of Solr rely on a local file system based Directory implementation - most notably, replication. This should all be changed to use the Directory and DirectoryFactory abstractions.
Other parts of the code that count on the local file system for making paths and getting file sizes should also be changed to use Directory and/or DirectoryFactory.
Original title: Replication should work with any Directory impl, not just local filesystem based Directories.
I've wanted to do this for a long time - there is no reason replication should not support any directory impl. This will let us use the mockdir for replication tests rather than having to force an FSDir and lose all the extra test checks and simulations. This will improve our testing around replication a lot, and allow custom Directory impls to be used on multi node Solr.
Expanded scope - full first class support for DirectoryFactory and Directory.
was:I've wanted to do this for a long time - there is no reason replication should not support any directory impl. This will let us use the mockdir for replication tests rather than having to force an FSDir and lose all the extra test checks and simulations. This will improve our testing around replication a lot, and allow custom Directory impls to be used on multi node Solr.
Summary: Make Directory and DirectoryFactory first class so that the majority of Solr's features work with any custom implementations. (was: Replication should work with any Directory impl, not just local filesystem based Directories.)
> Make Directory and DirectoryFactory first class so that the majority of Solr's features work with any custom implementations.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-3911
> URL: https://issues.apache.org/jira/browse/SOLR-3911
> Project: Solr
> Issue Type: Improvement
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-3911.patch, SOLR-3911.patch
>
>
> The biggest issue is that many parts of Solr rely on a local file system based Directory implementation - most notably, replication. This should all be changed to use the Directory and DirectoryFactory abstractions.
> Other parts of the code that count on the local file system for making paths and getting file sizes should also be changed to use Directory and/or DirectoryFactory.
> Original title: Replication should work with any Directory impl, not just local filesystem based Directories.
> I've wanted to do this for a long time - there is no reason replication should not support any directory impl. This will let us use the mockdir for replication tests rather than having to force an FSDir and lose all the extra test checks and simulations. This will improve our testing around replication a lot, and allow custom Directory impls to be used on multi node Solr.
> Expanded scope - full first class support for DirectoryFactory and Directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3911) Replication should work with any
Directory impl, not just local filesystem based Directories.
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-3911:
------------------------------
Attachment: SOLR-3911.patch
First draft:
All tests pass with a few caveats:
* Replication tests that replicate config files are currently disabled and it's not yet supported.
* Replication backup command test is disabled and not yet supported.
* There are a few nocommits around little issues to work out.
* Still need to add back local FileSystem optimizations - mostly doing rename for a move rather than copy from one Directory impl to another.
In pretty good shape though.
> Replication should work with any Directory impl, not just local filesystem based Directories.
> ---------------------------------------------------------------------------------------------
>
> Key: SOLR-3911
> URL: https://issues.apache.org/jira/browse/SOLR-3911
> Project: Solr
> Issue Type: Improvement
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-3911.patch
>
>
> I've wanted to do this for a long time - there is no reason replication should not support any directory impl. This will let us use the mockdir for replication tests rather than having to force an FSDir and lose all the extra test checks and simulations. This will improve our testing around replication a lot, and allow custom Directory impls to be used on multi node Solr.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3911) Make Directory and DirectoryFactory
first class so that the majority of Solr's features work with any custom
implementations.
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485471#comment-13485471 ]
Mark Miller commented on SOLR-3911:
-----------------------------------
I missed converting one of the prop files we write to using a Directory (logReplicationTimeAndConfFiles). This is the file with replication history and stats - obviously no testing for it :)
> Make Directory and DirectoryFactory first class so that the majority of Solr's features work with any custom implementations.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-3911
> URL: https://issues.apache.org/jira/browse/SOLR-3911
> Project: Solr
> Issue Type: Improvement
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-3911.patch, SOLR-3911.patch, SOLR-3911.patch
>
>
> The biggest issue is that many parts of Solr rely on a local file system based Directory implementation - most notably, replication. This should all be changed to use the Directory and DirectoryFactory abstractions.
> Other parts of the code that count on the local file system for making paths and getting file sizes should also be changed to use Directory and/or DirectoryFactory.
> Original title: Replication should work with any Directory impl, not just local filesystem based Directories.
> I've wanted to do this for a long time - there is no reason replication should not support any directory impl. This will let us use the mockdir for replication tests rather than having to force an FSDir and lose all the extra test checks and simulations. This will improve our testing around replication a lot, and allow custom Directory impls to be used on multi node Solr.
> Expanded scope - full first class support for DirectoryFactory and Directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3911) Replication should work with any
Directory impl, not just local filesystem based Directories.
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468175#comment-13468175 ]
Mark Miller commented on SOLR-3911:
-----------------------------------
Two other things not yet handled:
* I took out any use of lastModified.
* I have not yet impl'd getting the directory size from a Directory rather than the filesystem - should be simple enough, right now I just return 0.
> Replication should work with any Directory impl, not just local filesystem based Directories.
> ---------------------------------------------------------------------------------------------
>
> Key: SOLR-3911
> URL: https://issues.apache.org/jira/browse/SOLR-3911
> Project: Solr
> Issue Type: Improvement
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-3911.patch
>
>
> I've wanted to do this for a long time - there is no reason replication should not support any directory impl. This will let us use the mockdir for replication tests rather than having to force an FSDir and lose all the extra test checks and simulations. This will improve our testing around replication a lot, and allow custom Directory impls to be used on multi node Solr.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3911) Make Directory and DirectoryFactory
first class so that the majority of Solr's features work with any custom
implementations.
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472883#comment-13472883 ]
Mark Miller commented on SOLR-3911:
-----------------------------------
I was a little surprised, but even almost all of the solrcloud tests pass without forcing a file system based directory. I thought they probably should, but didn't think I'd end up so lucky.
Only one cloud test is failing when I don't force a local fs based dir for it - I'll try and look into why soon.
> Make Directory and DirectoryFactory first class so that the majority of Solr's features work with any custom implementations.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-3911
> URL: https://issues.apache.org/jira/browse/SOLR-3911
> Project: Solr
> Issue Type: Improvement
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-3911.patch, SOLR-3911.patch
>
>
> The biggest issue is that many parts of Solr rely on a local file system based Directory implementation - most notably, replication. This should all be changed to use the Directory and DirectoryFactory abstractions.
> Other parts of the code that count on the local file system for making paths and getting file sizes should also be changed to use Directory and/or DirectoryFactory.
> Original title: Replication should work with any Directory impl, not just local filesystem based Directories.
> I've wanted to do this for a long time - there is no reason replication should not support any directory impl. This will let us use the mockdir for replication tests rather than having to force an FSDir and lose all the extra test checks and simulations. This will improve our testing around replication a lot, and allow custom Directory impls to be used on multi node Solr.
> Expanded scope - full first class support for DirectoryFactory and Directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3911) Make Directory and DirectoryFactory
first class so that the majority of Solr's features work with any custom
implementations.
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472949#comment-13472949 ]
Mark Miller commented on SOLR-3911:
-----------------------------------
bq. Only one cloud test is failing when I don't force a local fs based dir for it
I see what this is about - RamDirectory does not keep it's contents over a restart and when a machine comes up and all of a sudden has no existing index but it still has it's old transaction log files, peer sync recovery than can succeed based on the transaction log, but the node will actually be out of sync.
> Make Directory and DirectoryFactory first class so that the majority of Solr's features work with any custom implementations.
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-3911
> URL: https://issues.apache.org/jira/browse/SOLR-3911
> Project: Solr
> Issue Type: Improvement
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-3911.patch, SOLR-3911.patch
>
>
> The biggest issue is that many parts of Solr rely on a local file system based Directory implementation - most notably, replication. This should all be changed to use the Directory and DirectoryFactory abstractions.
> Other parts of the code that count on the local file system for making paths and getting file sizes should also be changed to use Directory and/or DirectoryFactory.
> Original title: Replication should work with any Directory impl, not just local filesystem based Directories.
> I've wanted to do this for a long time - there is no reason replication should not support any directory impl. This will let us use the mockdir for replication tests rather than having to force an FSDir and lose all the extra test checks and simulations. This will improve our testing around replication a lot, and allow custom Directory impls to be used on multi node Solr.
> Expanded scope - full first class support for DirectoryFactory and Directory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3911) Replication should work with any
Directory impl, not just local filesystem based Directories.
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-3911:
------------------------------
Attachment: SOLR-3911.patch
New patch.
All tests are now uncommented and pass.
Still some more cleanup and checking to do. Especially some hard thinking around where we use File.getCanonical and File.getAbsolute path. There are possibly issues our test cases are not catching.
Still have a mkdir issue to address properly - right now I just special case for a couple fs based Directory impls.
sizeOfIndexDirectory now works based on Directory.
Still have not optimized rename/move for fs based Directory impls.
Looking good though - this is fairly close.
> Replication should work with any Directory impl, not just local filesystem based Directories.
> ---------------------------------------------------------------------------------------------
>
> Key: SOLR-3911
> URL: https://issues.apache.org/jira/browse/SOLR-3911
> Project: Solr
> Issue Type: Improvement
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-3911.patch, SOLR-3911.patch
>
>
> I've wanted to do this for a long time - there is no reason replication should not support any directory impl. This will let us use the mockdir for replication tests rather than having to force an FSDir and lose all the extra test checks and simulations. This will improve our testing around replication a lot, and allow custom Directory impls to be used on multi node Solr.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3911) Replication should work with any
Directory impl, not just local filesystem based Directories.
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472095#comment-13472095 ]
Mark Miller commented on SOLR-3911:
-----------------------------------
Yeah, sure.
I've got the conf tests passing tonight. Just have to do backup and these other clean up issues.
> Replication should work with any Directory impl, not just local filesystem based Directories.
> ---------------------------------------------------------------------------------------------
>
> Key: SOLR-3911
> URL: https://issues.apache.org/jira/browse/SOLR-3911
> Project: Solr
> Issue Type: Improvement
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.1, 5.0
>
> Attachments: SOLR-3911.patch
>
>
> I've wanted to do this for a long time - there is no reason replication should not support any directory impl. This will let us use the mockdir for replication tests rather than having to force an FSDir and lose all the extra test checks and simulations. This will improve our testing around replication a lot, and allow custom Directory impls to be used on multi node Solr.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org