You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2010/09/17 19:52:37 UTC
[jira] Created: (LUCENE-2650) improve windows defaults in
FSDirectory
improve windows defaults in FSDirectory
---------------------------------------
Key: LUCENE-2650
URL: https://issues.apache.org/jira/browse/LUCENE-2650
Project: Lucene - Java
Issue Type: Improvement
Components: Store
Affects Versions: 4.0
Reporter: Robert Muir
Assignee: Robert Muir
Fix For: 4.0
Currently windows defaults to SimpleFSDirectory, but this is a problem due to the synchronization.
I have been benchmarking queries *sequentially* and was pretty surprised at how much faster
MMapDirectory is, for example for cases that do many seeks.
I think we should change the defaults for windows as such:
if (WINDOWS and UNMAP_SUPPORTED and 64-bit)
use MMapDirectory
else
use SimpleFSDirectory
I think we should just consider doing this for 4.0 only and see how it goes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2650) improve windows defaults in
FSDirectory
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910760#action_12910760 ]
Yonik Seeley commented on LUCENE-2650:
--------------------------------------
bq. I think we should just consider doing this for 4.0 only and see how it goes.
+1, that does seem to be the best option for windows servers at this point.
> improve windows defaults in FSDirectory
> ---------------------------------------
>
> Key: LUCENE-2650
> URL: https://issues.apache.org/jira/browse/LUCENE-2650
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 4.0
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 4.0
>
> Attachments: LUCENE-2650.patch
>
>
> Currently windows defaults to SimpleFSDirectory, but this is a problem due to the synchronization.
> I have been benchmarking queries *sequentially* and was pretty surprised at how much faster
> MMapDirectory is, for example for cases that do many seeks.
> I think we should change the defaults for windows as such:
> if (WINDOWS and UNMAP_SUPPORTED and 64-bit)
> use MMapDirectory
> else
> use SimpleFSDirectory
> I think we should just consider doing this for 4.0 only and see how it goes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2650) improve windows defaults in
FSDirectory
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir resolved LUCENE-2650.
---------------------------------
Resolution: Fixed
Committed revision 999409 to trunk.
> improve windows defaults in FSDirectory
> ---------------------------------------
>
> Key: LUCENE-2650
> URL: https://issues.apache.org/jira/browse/LUCENE-2650
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 4.0
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 4.0
>
> Attachments: LUCENE-2650.patch, LUCENE-2650.patch
>
>
> Currently windows defaults to SimpleFSDirectory, but this is a problem due to the synchronization.
> I have been benchmarking queries *sequentially* and was pretty surprised at how much faster
> MMapDirectory is, for example for cases that do many seeks.
> I think we should change the defaults for windows as such:
> if (WINDOWS and UNMAP_SUPPORTED and 64-bit)
> use MMapDirectory
> else
> use SimpleFSDirectory
> I think we should just consider doing this for 4.0 only and see how it goes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2650) improve windows defaults in
FSDirectory
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2650:
--------------------------------
Attachment: LUCENE-2650.patch
Here's the patch:
* sets the defaults as described in the issue
* turns on unmap by default if its possible
* when unmapping inputs, we should be careful to never allow the region to be accessed.
so in close(), we ensure that clones, too set their buffer to null.
and in clone() we add a friendlier exception rather than NPE.
> improve windows defaults in FSDirectory
> ---------------------------------------
>
> Key: LUCENE-2650
> URL: https://issues.apache.org/jira/browse/LUCENE-2650
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 4.0
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 4.0
>
> Attachments: LUCENE-2650.patch
>
>
> Currently windows defaults to SimpleFSDirectory, but this is a problem due to the synchronization.
> I have been benchmarking queries *sequentially* and was pretty surprised at how much faster
> MMapDirectory is, for example for cases that do many seeks.
> I think we should change the defaults for windows as such:
> if (WINDOWS and UNMAP_SUPPORTED and 64-bit)
> use MMapDirectory
> else
> use SimpleFSDirectory
> I think we should just consider doing this for 4.0 only and see how it goes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2650) improve windows defaults in
FSDirectory
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2650:
----------------------------------
Attachment: LUCENE-2651.patch
I completed the patch, I missed the extra check in clone().
> improve windows defaults in FSDirectory
> ---------------------------------------
>
> Key: LUCENE-2650
> URL: https://issues.apache.org/jira/browse/LUCENE-2650
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 4.0
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 4.0
>
> Attachments: LUCENE-2650.patch, LUCENE-2650.patch, LUCENE-2651.patch
>
>
> Currently windows defaults to SimpleFSDirectory, but this is a problem due to the synchronization.
> I have been benchmarking queries *sequentially* and was pretty surprised at how much faster
> MMapDirectory is, for example for cases that do many seeks.
> I think we should change the defaults for windows as such:
> if (WINDOWS and UNMAP_SUPPORTED and 64-bit)
> use MMapDirectory
> else
> use SimpleFSDirectory
> I think we should just consider doing this for 4.0 only and see how it goes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2650) improve windows defaults in
FSDirectory
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12911064#action_12911064 ]
Robert Muir commented on LUCENE-2650:
-------------------------------------
by the way, i ran all tests with -Dtests.directory=MMapDirectory under both Oracle/BEA Jrockit and IBM (although this one: 32 bit)
both these support the unmap hack and all lucene/solr tests pass (except for an unrelated thai issue: LUCENE-2653)
so i'd like to commit this to trunk only in a few days if no one objects.
> improve windows defaults in FSDirectory
> ---------------------------------------
>
> Key: LUCENE-2650
> URL: https://issues.apache.org/jira/browse/LUCENE-2650
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 4.0
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 4.0
>
> Attachments: LUCENE-2650.patch, LUCENE-2650.patch
>
>
> Currently windows defaults to SimpleFSDirectory, but this is a problem due to the synchronization.
> I have been benchmarking queries *sequentially* and was pretty surprised at how much faster
> MMapDirectory is, for example for cases that do many seeks.
> I think we should change the defaults for windows as such:
> if (WINDOWS and UNMAP_SUPPORTED and 64-bit)
> use MMapDirectory
> else
> use SimpleFSDirectory
> I think we should just consider doing this for 4.0 only and see how it goes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2650) improve windows defaults in
FSDirectory
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12912838#action_12912838 ]
Robert Muir commented on LUCENE-2650:
-------------------------------------
I'm going to add the extra safety here for cloned mmapindexinputs as a separate commit
from changing the defaults (in case we have to revert the defaults).
Its also good to backport (unlike the defaults)
> improve windows defaults in FSDirectory
> ---------------------------------------
>
> Key: LUCENE-2650
> URL: https://issues.apache.org/jira/browse/LUCENE-2650
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 4.0
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 4.0
>
> Attachments: LUCENE-2650.patch, LUCENE-2650.patch
>
>
> Currently windows defaults to SimpleFSDirectory, but this is a problem due to the synchronization.
> I have been benchmarking queries *sequentially* and was pretty surprised at how much faster
> MMapDirectory is, for example for cases that do many seeks.
> I think we should change the defaults for windows as such:
> if (WINDOWS and UNMAP_SUPPORTED and 64-bit)
> use MMapDirectory
> else
> use SimpleFSDirectory
> I think we should just consider doing this for 4.0 only and see how it goes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2650) improve windows defaults in
FSDirectory
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910695#action_12910695 ]
Robert Muir commented on LUCENE-2650:
-------------------------------------
by the way, all lucene/solr tests pass on 64-bit windows with -Dtests.directory=FSDirectory
> improve windows defaults in FSDirectory
> ---------------------------------------
>
> Key: LUCENE-2650
> URL: https://issues.apache.org/jira/browse/LUCENE-2650
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 4.0
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 4.0
>
> Attachments: LUCENE-2650.patch
>
>
> Currently windows defaults to SimpleFSDirectory, but this is a problem due to the synchronization.
> I have been benchmarking queries *sequentially* and was pretty surprised at how much faster
> MMapDirectory is, for example for cases that do many seeks.
> I think we should change the defaults for windows as such:
> if (WINDOWS and UNMAP_SUPPORTED and 64-bit)
> use MMapDirectory
> else
> use SimpleFSDirectory
> I think we should just consider doing this for 4.0 only and see how it goes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2650) improve windows defaults in
FSDirectory
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2650:
----------------------------------
Attachment: (was: LUCENE-2651.patch)
> improve windows defaults in FSDirectory
> ---------------------------------------
>
> Key: LUCENE-2650
> URL: https://issues.apache.org/jira/browse/LUCENE-2650
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 4.0
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 4.0
>
> Attachments: LUCENE-2650.patch, LUCENE-2650.patch
>
>
> Currently windows defaults to SimpleFSDirectory, but this is a problem due to the synchronization.
> I have been benchmarking queries *sequentially* and was pretty surprised at how much faster
> MMapDirectory is, for example for cases that do many seeks.
> I think we should change the defaults for windows as such:
> if (WINDOWS and UNMAP_SUPPORTED and 64-bit)
> use MMapDirectory
> else
> use SimpleFSDirectory
> I think we should just consider doing this for 4.0 only and see how it goes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2650) improve windows defaults in
FSDirectory
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2650:
--------------------------------
Attachment: LUCENE-2650.patch
same patch, only i added the same safety (ensuring clones null their buffers) to the MultiMMapII that I added to the MMapII.
> improve windows defaults in FSDirectory
> ---------------------------------------
>
> Key: LUCENE-2650
> URL: https://issues.apache.org/jira/browse/LUCENE-2650
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 4.0
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 4.0
>
> Attachments: LUCENE-2650.patch, LUCENE-2650.patch
>
>
> Currently windows defaults to SimpleFSDirectory, but this is a problem due to the synchronization.
> I have been benchmarking queries *sequentially* and was pretty surprised at how much faster
> MMapDirectory is, for example for cases that do many seeks.
> I think we should change the defaults for windows as such:
> if (WINDOWS and UNMAP_SUPPORTED and 64-bit)
> use MMapDirectory
> else
> use SimpleFSDirectory
> I think we should just consider doing this for 4.0 only and see how it goes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2650) improve windows defaults in
FSDirectory
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2650:
----------------------------------
Comment: was deleted
(was: I completed the patch, I missed the extra check in clone().)
> improve windows defaults in FSDirectory
> ---------------------------------------
>
> Key: LUCENE-2650
> URL: https://issues.apache.org/jira/browse/LUCENE-2650
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 4.0
> Reporter: Robert Muir
> Assignee: Robert Muir
> Fix For: 4.0
>
> Attachments: LUCENE-2650.patch, LUCENE-2650.patch
>
>
> Currently windows defaults to SimpleFSDirectory, but this is a problem due to the synchronization.
> I have been benchmarking queries *sequentially* and was pretty surprised at how much faster
> MMapDirectory is, for example for cases that do many seeks.
> I think we should change the defaults for windows as such:
> if (WINDOWS and UNMAP_SUPPORTED and 64-bit)
> use MMapDirectory
> else
> use SimpleFSDirectory
> I think we should just consider doing this for 4.0 only and see how it goes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org