You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Luca Cavanna (Created) (JIRA)" <ji...@apache.org> on 2011/11/22 15:14:40 UTC
[jira] [Created] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Choose a specific Directory implementation running the CheckIndex main
----------------------------------------------------------------------
Key: LUCENE-3586
URL: https://issues.apache.org/jira/browse/LUCENE-3586
Project: Lucene - Java
Issue Type: Improvement
Reporter: Luca Cavanna
Priority: Minor
It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
What about an additional main parameter?
In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
Should we also consider to use a FileSwitchDirectory?
I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Luca Cavanna (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luca Cavanna reassigned LUCENE-3586:
------------------------------------
Assignee: Luca Cavanna
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158357#comment-13158357 ]
Michael McCandless commented on LUCENE-3586:
--------------------------------------------
Hmm, I don't think we should add an enum to FSDir here? Can we simply accept the class name and then just load that class (maybe prefixing oal.store so user doesn't have to type that all the time)?
Also, can we make it a hard error if the specified name isn't recognized? (Instead of silently falling back to FSDir.open).
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
> Attachments: LUCENE-3586.patch
>
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Luca Cavanna (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155195#comment-13155195 ]
Luca Cavanna commented on LUCENE-3586:
--------------------------------------
Cool! Thanks, tomorrow I'll add a patch for this.
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Luca Cavanna (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168545#comment-13168545 ]
Luca Cavanna commented on LUCENE-3586:
--------------------------------------
Cool, thanks to you Michael!
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3586.patch, LUCENE-3586.patch, LUCENE-3586.patch, LUCENE-3586.patch
>
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Luca Cavanna (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168290#comment-13168290 ]
Luca Cavanna commented on LUCENE-3586:
--------------------------------------
Hi Michael,
{quote}
Actually I don't like all the generics required in CommandLineUtil solely to support LTC's case; if not for LTC the util would be much simpler (just one method taking a String class name/path and returning a Directory instance).
{quote}
I see. But I think we were on the right track. Have a look at the last version, I kept the merged code removing that method with generics. If you don't like even this version I'll go back, no problem at all!
{quote}
Can you add the lost " // for javadocs" comment on one of CheckIndex's imports?
{quote}
Ops, I restored it.
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
> Attachments: LUCENE-3586.patch, LUCENE-3586.patch, LUCENE-3586.patch, LUCENE-3586.patch
>
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Luca Cavanna (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155894#comment-13155894 ]
Luca Cavanna commented on LUCENE-3586:
--------------------------------------
Hi Uwe,
I'm working with lucene 3.4 and jvm 1.6.0_23 64 bit. I have ulimit unlimited and /proc/sys/vm/max_map_count 65530. The index size is 76GB, I have problems with just one segment, the first and definitely the bigger one, on which the fdt is 22GB, tvf is 13GB, prx is 4.2GB, frq is 2.8GB, and so on (others are under 1GB). I tried to run the checkindex with -Xmx512m and -Xms512m. Here is my output:
Segments file=segments_kj numSegments=31 version=3.4 format=FORMAT_3_1 [Lucene 3.1+]
1 of 31: name=_1iz docCount=3186709
compound=false
hasProx=true
numFiles=12
size (MB)=41,902.362
diagnostics = {optimize=true, mergeFactor=23, os.version=2.6.27.59-1POi-x86_64, os=Linux, lucene.version=3.4.0 1167142 - mike - 2011-09-09 09:02:09, source=merge, os.arch=amd64, java.version=1.6.0_23, java.vendor=Sun Microsystems Inc.}
has deletions [delFileName=_1iz_r.del]
test: open reader.........FAILED
Thanks, and please let me know if you have any questions.
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155235#comment-13155235 ]
Uwe Schindler commented on LUCENE-3586:
---------------------------------------
About the issue at all:
we should maybe have this parameter to all command line tools (e.g. IndexUpgrader,...). It would be good to have some general command line parser for those comand line tools that work on indexes, so we don't have the same/similar code several times.
For me much more interesting is the problem you have with MMAP? Whats you Java VM / Lucene version and what's your complete command line to CheckIndex? What architecture are you using and which OS? Changing the chunk size is not needed in most cases, if it needs to we need complete configuration details to optimize auto detection.
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168528#comment-13168528 ]
Michael McCandless commented on LUCENE-3586:
--------------------------------------------
New patch looks great Luca! I'll commit & backport shortly...
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
> Attachments: LUCENE-3586.patch, LUCENE-3586.patch, LUCENE-3586.patch, LUCENE-3586.patch
>
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Simon Willnauer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155187#comment-13155187 ]
Simon Willnauer commented on LUCENE-3586:
-----------------------------------------
Luca, I made you a contributor for Lucene & Solr (JIRA Role) so you can assign issues to you.
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Priority: Minor
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Luca Cavanna (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luca Cavanna updated LUCENE-3586:
---------------------------------
Attachment: LUCENE-3586.patch
Here is the first version of my patch, please let me know your comments.
I tried to do something more generic for command line tools and less "static" but I found myself reinventing the wheel, writing stuff similar to commons cli, so I gave up.
I basically added a new option called -dir-impl to CheckIndex, IndexReader and IndexUpgrader (please let me know if I forgot some other tools). The new option can have the following values (case insensitive): mmap, nio or simple. Any other value is ignored and considered as default (FSDirectory.open).
I noticed a finally block used to close the directory inside IndexReader, but it's missing on the other two classes, wouldn't it be better to add it to CheckIndex and IndexUpgrader?
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
> Attachments: LUCENE-3586.patch
>
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160102#comment-13160102 ]
Michael McCandless commented on LUCENE-3586:
--------------------------------------------
I think just load the classes by name via reflection? This way if I have my own external Dir impl somewhere I can also have CheckIndex use that...
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
> Attachments: LUCENE-3586.patch
>
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Luca Cavanna (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luca Cavanna updated LUCENE-3586:
---------------------------------
Attachment: LUCENE-3586.patch
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
> Attachments: LUCENE-3586.patch, LUCENE-3586.patch, LUCENE-3586.patch, LUCENE-3586.patch
>
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-3586) Choose a specific
Directory implementation running the CheckIndex main
Posted by "Luca Cavanna (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155894#comment-13155894 ]
Luca Cavanna edited comment on LUCENE-3586 at 11/23/11 2:23 PM:
----------------------------------------------------------------
Hi Uwe,
I'm working with lucene 3.4 and jvm 1.6.0_23 64 bit. I have ulimit unlimited and /proc/sys/vm/max_map_count 65530. The index size is 76GB, I have problems with just one segment, the first and definitely the biggest one, on which the fdt is 22GB, tvf is 13GB, prx is 4.2GB, frq is 2.8GB, and so on (others are under 1GB). I tried to run the checkindex with -Xmx512m and -Xms512m. Here is my output:
Segments file=segments_kj numSegments=31 version=3.4 format=FORMAT_3_1 [Lucene 3.1+]
1 of 31: name=_1iz docCount=3186709
compound=false
hasProx=true
numFiles=12
size (MB)=41,902.362
diagnostics = {optimize=true, mergeFactor=23, os.version=2.6.27.59-1POi-x86_64, os=Linux, lucene.version=3.4.0 1167142 - mike - 2011-09-09 09:02:09, source=merge, os.arch=amd64, java.version=1.6.0_23, java.vendor=Sun Microsystems Inc.}
has deletions [delFileName=_1iz_r.del]
test: open reader.........FAILED
Thanks, and please let me know if you have any questions.
was (Author: lucacavanna):
Hi Uwe,
I'm working with lucene 3.4 and jvm 1.6.0_23 64 bit. I have ulimit unlimited and /proc/sys/vm/max_map_count 65530. The index size is 76GB, I have problems with just one segment, the first and definitely the bigger one, on which the fdt is 22GB, tvf is 13GB, prx is 4.2GB, frq is 2.8GB, and so on (others are under 1GB). I tried to run the checkindex with -Xmx512m and -Xms512m. Here is my output:
Segments file=segments_kj numSegments=31 version=3.4 format=FORMAT_3_1 [Lucene 3.1+]
1 of 31: name=_1iz docCount=3186709
compound=false
hasProx=true
numFiles=12
size (MB)=41,902.362
diagnostics = {optimize=true, mergeFactor=23, os.version=2.6.27.59-1POi-x86_64, os=Linux, lucene.version=3.4.0 1167142 - mike - 2011-09-09 09:02:09, source=merge, os.arch=amd64, java.version=1.6.0_23, java.vendor=Sun Microsystems Inc.}
has deletions [delFileName=_1iz_r.del]
test: open reader.........FAILED
Thanks, and please let me know if you have any questions.
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Simon Willnauer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155184#comment-13155184 ]
Simon Willnauer commented on LUCENE-3586:
-----------------------------------------
I think this makes sense though. sometimes like in your usecase MMap just fails with OOM and since we create the directory instance based on constants like 64bit platform, OS etc. we could make this optional. another option would be to simply use NIOFSDirectory no matter where we are. I don't think it is necessary to use FileSwitchDirectory for CheckIndex. Maybe you can add a commandline option and get a specific Dir impl if specified? NIO, MMAP & SimpleFS should be enough and use FSDirectory.open by default.
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Priority: Minor
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Luca Cavanna (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158309#comment-13158309 ]
Luca Cavanna commented on LUCENE-3586:
--------------------------------------
Hey Simon,
looks better! I thought I wasn't allowed to edit FSDirectory just for a new command line option :)
I suppose we don't need also FSDirectory#open(File, LockFactory, Type) because we can't specify the LockFactory from command line, right?
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
> Attachments: LUCENE-3586.patch
>
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Luca Cavanna (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luca Cavanna updated LUCENE-3586:
---------------------------------
Attachment: LUCENE-3586.patch
Good eye! Thanks Michael for your feedback.
I attached a new patch version. The merge hasn't been so easy since LTC needs sometimes a Directory, while other times a FSDirectory. Furthermore commmand line utilities don't need a fallback while tests do..before the tests we even create the directory on file system etc.
Let me know what you think!
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
> Attachments: LUCENE-3586.patch, LUCENE-3586.patch, LUCENE-3586.patch
>
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Luca Cavanna (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luca Cavanna updated LUCENE-3586:
---------------------------------
Attachment: LUCENE-3586.patch
New patch against trunk according to Michael's hints.
It's now possible to use external FSDirectory implementations. The package oal.store is used if no package is specified. This isn't good if someone has the FSDirectory implementation in the default package, but I'm not sure if this case is worth a fall back. Please, let me know what you think.
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
> Attachments: LUCENE-3586.patch, LUCENE-3586.patch
>
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168324#comment-13168324 ]
Uwe Schindler commented on LUCENE-3586:
---------------------------------------
What is the problem with the generics code? I like it :-)
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
> Attachments: LUCENE-3586.patch, LUCENE-3586.patch, LUCENE-3586.patch, LUCENE-3586.patch
>
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-3586) Choose a specific
Directory implementation running the CheckIndex main
Posted by "Uwe Schindler (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155235#comment-13155235 ]
Uwe Schindler edited comment on LUCENE-3586 at 11/22/11 4:33 PM:
-----------------------------------------------------------------
About the issue at all:
we should maybe have this parameter to all command line tools (e.g. IndexUpgrader,...). It would be good to have some general command line parser for those comand line tools that work on indexes, so we don't have the same/similar code several times.
For me much more interesting is the problem you have with MMAP? Whats you Java VM / Lucene version and what's your complete command line to CheckIndex? How big is the index? What architecture are you using and which OS? Changing the chunk size is not needed in most cases, if it needs to we need complete configuration details to optimize auto detection.
was (Author: thetaphi):
About the issue at all:
we should maybe have this parameter to all command line tools (e.g. IndexUpgrader,...). It would be good to have some general command line parser for those comand line tools that work on indexes, so we don't have the same/similar code several times.
For me much more interesting is the problem you have with MMAP? Whats you Java VM / Lucene version and what's your complete command line to CheckIndex? What architecture are you using and which OS? Changing the chunk size is not needed in most cases, if it needs to we need complete configuration details to optimize auto detection.
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167672#comment-13167672 ]
Michael McCandless commented on LUCENE-3586:
--------------------------------------------
Hi Luca,
Actually I don't like all the generics required in CommandLineUtil solely to support LTC's case; if not for LTC the util would be much simpler (just one method taking a String class name/path and returning a Directory instance).
So.... I think we should go back, and leave LTC with its current (private) code for finding the FS/Directory class/instance, and keep CommandLineUtil simple? Sorry!
Otherwise the patch looks great! Can you add the lost " // for javadocs" comment on one of CheckIndex's imports? Thanks.
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
> Attachments: LUCENE-3586.patch, LUCENE-3586.patch, LUCENE-3586.patch
>
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Luca Cavanna (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155285#comment-13155285 ]
Luca Cavanna commented on LUCENE-3586:
--------------------------------------
Hi Uwe, hi all,
you're right, I know (virtually) almost every people working at Solr/Lucene, but you of course don't know me!
I'm Luca, I'm a software developer actually working at Dutchworks in Amsterdam within the enterprise search team. You certainly know my colleague Martijn, Solr/Lucene committer. I really like the open source world and the solr/lucene project. These are my contributions so far: SOLR-1499 and SOLR-2591. Just a few, but I would like to contribute as much as I can. Please,don't hesitate to ask if you have any questions.
I assigned the issue to myself because I thought I was gonna work on it, but let me know if I made some mistakes.
Tomorrow I'll add more details about the MMAP and the OOM.
Thanks for your help and for your point of view about the issue, it would be great to do something more generic than I initially thought, I'll have a look at the code.
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Luca Cavanna (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157226#comment-13157226 ]
Luca Cavanna commented on LUCENE-3586:
--------------------------------------
Forgot to mention that the patch applies to the 3x branch. After your comments, and maybe some changes, I'll attach a patch for the trunk as well.
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
> Attachments: LUCENE-3586.patch
>
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166965#comment-13166965 ]
Michael McCandless commented on LUCENE-3586:
--------------------------------------------
Thanks Luca; I think this patch is close!
One thing I noticed is that LuceneTestCase.java (under
src/test-framework/java/org/apache/lucene/util), has a method
newDirectoryImpl doing almost the same thing as the new static method
here. Maybe merge these two (have LTC call the new one, but leave that
random logic in LTC)?
Separately, can we put the static method in oal.util? Maybe in a new
file, CommandLineUtil.java or something?
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
> Attachments: LUCENE-3586.patch, LUCENE-3586.patch
>
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Simon Willnauer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158296#comment-13158296 ]
Simon Willnauer commented on LUCENE-3586:
-----------------------------------------
hey luca, you patch looks good. I wonder if we should maybe add an enum to FSDirectory like
{code}
enum Type {
NIOFS,
MMAP,
SIMPLE;
}
{code}
and add FSDirectory#open(File, Type). Just an idea, this would make the CommandlineUtils obsolete and I think it could be useful... not sure what others think... uwe?
bq. Forgot to mention that the patch applies to the 3x branch
usually we (I) only provide a patch against trunk since we do a merge when we backport so a patch doesn't help very often. So it'd be better if you'd use trunk to create patches.
simon
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
> Attachments: LUCENE-3586.patch
>
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Michael McCandless (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-3586.
----------------------------------------
Resolution: Fixed
Fix Version/s: 4.0
3.6
Thanks Luca!
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3586.patch, LUCENE-3586.patch, LUCENE-3586.patch, LUCENE-3586.patch
>
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Luca Cavanna (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159894#comment-13159894 ]
Luca Cavanna commented on LUCENE-3586:
--------------------------------------
{quote}
Hmm, I don't think we should add an enum to FSDir here? Can we simply accept the class name and then just load that class (maybe prefixing oal.store so user doesn't have to type that all the time)?
Also, can we make it a hard error if the specified name isn't recognized? (Instead of silently falling back to FSDir.open).
{quote}
That's fine as well. Just a little bit longer than writing NIOFS, MMAP or SIMPLE, but I guess it doesn't matter. Mike, do you mean to load the class using reflection or compare the input string to those three class names?
Any other opinion?
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
> Attachments: LUCENE-3586.patch
>
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3586) Choose a specific Directory
implementation running the CheckIndex main
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155233#comment-13155233 ]
Uwe Schindler commented on LUCENE-3586:
---------------------------------------
Hi Simon, hi Luca,
Maybe we get some introduction and background from Luca, because in general issues are assigned to committers/contributors that have already done work for Lucene/Solr?
Uwe
> Choose a specific Directory implementation running the CheckIndex main
> ----------------------------------------------------------------------
>
> Key: LUCENE-3586
> URL: https://issues.apache.org/jira/browse/LUCENE-3586
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Luca Cavanna
> Assignee: Luca Cavanna
> Priority: Minor
>
> It should be possible to choose a specific Directory implementation to use during the CheckIndex process when we run it from its main.
> What about an additional main parameter?
> In fact, I'm experiencing some problems with MMapDirectory working with a big segment, and after some failed attempts playing with maxChunkSize, I decided to switch to another FSDirectory implementation but I needed to do that on my own main.
> Should we also consider to use a FileSwitchDirectory?
> I'm willing to contribute, could you please let me know your thoughts about it?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org