You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Uwe Schindler (JIRA)" <ji...@apache.org> on 2009/07/13 16:13:14 UTC
[jira] Created: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user
configureable to support chunking the index files in smaller parts
Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
---------------------------------------------------------------------------------------------------
Key: LUCENE-1741
URL: https://issues.apache.org/jira/browse/LUCENE-1741
Project: Lucene - Java
Issue Type: Improvement
Affects Versions: 2.9
Reporter: Uwe Schindler
Assignee: Uwe Schindler
Priority: Minor
Fix For: 3.1
This is a followup for java-user thred: http://www.lucidimagination.com/search/document/9ba9137bb5d8cb78/oom_with_2_9#9bf3b5b8f3b1fb9b
It is easy to implement, just add a setter method for this parameter to MMapDir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user
configureable to support chunking the index files in smaller parts
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730745#action_12730745 ]
Michael McCandless commented on LUCENE-1741:
--------------------------------------------
Patch looks good; thanks Uwe.
> Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
> ---------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1741
> URL: https://issues.apache.org/jira/browse/LUCENE-1741
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1741.patch, LUCENE-1741.patch, LUCENE-1741.patch
>
>
> This is a followup for java-user thred: http://www.lucidimagination.com/search/document/9ba9137bb5d8cb78/oom_with_2_9#9bf3b5b8f3b1fb9b
> It is easy to implement, just add a setter method for this parameter to MMapDir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user
configureable to support chunking the index files in smaller parts
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730543#action_12730543 ]
Uwe Schindler commented on LUCENE-1741:
---------------------------------------
Eks Dev wrote in java-dev:
bq. I have no test data which size is good, it is just trying out
Sure, for this you need bad OS and large index, you are not as lucky as I am to have it :)
Anyhow, I would argument against default value. An algorithm is quite simple, if you hit OOM on map(), reduce this value until it fits :)
no need to touch it if it works...
> Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
> ---------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1741
> URL: https://issues.apache.org/jira/browse/LUCENE-1741
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1741.patch, LUCENE-1741.patch
>
>
> This is a followup for java-user thred: http://www.lucidimagination.com/search/document/9ba9137bb5d8cb78/oom_with_2_9#9bf3b5b8f3b1fb9b
> It is easy to implement, just add a setter method for this parameter to MMapDir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user
configureable to support chunking the index files in smaller parts
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-1741:
----------------------------------
Attachment: LUCENE-1741.patch
Patch that allows configuration of chunk size. I will commit in the evening (MEZ).
> Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
> ---------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1741
> URL: https://issues.apache.org/jira/browse/LUCENE-1741
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-1741.patch
>
>
> This is a followup for java-user thred: http://www.lucidimagination.com/search/document/9ba9137bb5d8cb78/oom_with_2_9#9bf3b5b8f3b1fb9b
> It is easy to implement, just add a setter method for this parameter to MMapDir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user
configureable to support chunking the index files in smaller parts
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730378#action_12730378 ]
Michael McCandless commented on LUCENE-1741:
--------------------------------------------
Patch looks good!
> Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
> ---------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1741
> URL: https://issues.apache.org/jira/browse/LUCENE-1741
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1741.patch, LUCENE-1741.patch
>
>
> This is a followup for java-user thred: http://www.lucidimagination.com/search/document/9ba9137bb5d8cb78/oom_with_2_9#9bf3b5b8f3b1fb9b
> It is easy to implement, just add a setter method for this parameter to MMapDir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user
configureable to support chunking the index files in smaller parts
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730368#action_12730368 ]
Michael McCandless commented on LUCENE-1741:
--------------------------------------------
Should we default the chunking size to something smaller (128 MB?) on 32 bit JRE?
> Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
> ---------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1741
> URL: https://issues.apache.org/jira/browse/LUCENE-1741
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1741.patch
>
>
> This is a followup for java-user thred: http://www.lucidimagination.com/search/document/9ba9137bb5d8cb78/oom_with_2_9#9bf3b5b8f3b1fb9b
> It is easy to implement, just add a setter method for this parameter to MMapDir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user
configureable to support chunking the index files in smaller parts
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730580#action_12730580 ]
Michael McCandless commented on LUCENE-1741:
--------------------------------------------
I'd be more comfortable w/ 256 MB (or, smaller); I think fragmentation could easily cause 512MB to give the false OOM. I don't think we'll see real perf costs from buffer switching unless chunk size is very small (eg < 1 MB).
In any event, Uwe can you add to the javadocs describing this false OOM problem and what to do if you hit it?
> Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
> ---------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1741
> URL: https://issues.apache.org/jira/browse/LUCENE-1741
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1741.patch, LUCENE-1741.patch
>
>
> This is a followup for java-user thred: http://www.lucidimagination.com/search/document/9ba9137bb5d8cb78/oom_with_2_9#9bf3b5b8f3b1fb9b
> It is easy to implement, just add a setter method for this parameter to MMapDir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user
configureable to support chunking the index files in smaller parts
Posted by "Eks Dev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730560#action_12730560 ]
Eks Dev commented on LUCENE-1741:
---------------------------------
Uwe, you convinced me, I looked at the code, and indeed, no performance penalty for this.
what helped me was 1.1G... (I've tried to find maximum); Max file size is 1.4G ... but 1.1 is just OS coincidence, no magic about it.
I guess 512mb makes a good value, if memory is so fragmented that you cannot allocate 0.5G, you are definitely having some other problems around. We are taliking here about VM memory, and even on windows having 512Mb in block is not an issue (or better said, I have never seen problems with this value).
@Paul: It is misunderstanding, my "algorithm" was meant to be manual... no catching OOM and retry (I've burned my fingers already on catching RuntimeException, do only when absolutely desperate :). Uwe made this value user settable anyhow.
Thanks Uwe!
> Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
> ---------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1741
> URL: https://issues.apache.org/jira/browse/LUCENE-1741
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1741.patch, LUCENE-1741.patch
>
>
> This is a followup for java-user thred: http://www.lucidimagination.com/search/document/9ba9137bb5d8cb78/oom_with_2_9#9bf3b5b8f3b1fb9b
> It is easy to implement, just add a setter method for this parameter to MMapDir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Updated: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
Posted by eks dev <ek...@yahoo.co.uk>.
>I have no test data which size is good, it is just trying out
Sure, for this you need bad OS and large index, you are not as lucky as I am to have it :)
Anyhow, I would argument against default value. An algorithm is quite simple, if you hit OOM on map(), reduce this value until it fits :)
no need to touch it if it works...
----- Original Message ----
> From: Uwe Schindler (JIRA) <ji...@apache.org>
> To: java-dev@lucene.apache.org
> Sent: Monday, 13 July, 2009 17:21:15
> Subject: [jira] Updated: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
>
>
> [
> https://issues.apache.org/jira/browse/LUCENE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> ]
>
> Uwe Schindler updated LUCENE-1741:
> ----------------------------------
>
> Attachment: LUCENE-1741.patch
>
> Attached is a patch using the JRE_IS_64BIT in Constants. I set the default to
> 256 MiBytes (128 seems to small for large indexes, if the index is e.g. about
> 1.5 GiBytes, you would get 6 junks.
>
> I have no test data which size is good, it is just trying out (and depends e.g.
> on how often you reboot Windows, as Eks said).
>
> > Make MMapDirectory.MAX_BBUF user configureable to support chunking the index
> files in smaller parts
> >
> ---------------------------------------------------------------------------------------------------
> >
> > Key: LUCENE-1741
> > URL: https://issues.apache.org/jira/browse/LUCENE-1741
> > Project: Lucene - Java
> > Issue Type: Improvement
> > Affects Versions: 2.9
> > Reporter: Uwe Schindler
> > Assignee: Uwe Schindler
> > Priority: Minor
> > Fix For: 2.9
> >
> > Attachments: LUCENE-1741.patch, LUCENE-1741.patch
> >
> >
> > This is a followup for java-user thred:
> http://www.lucidimagination.com/search/document/9ba9137bb5d8cb78/oom_with_2_9#9bf3b5b8f3b1fb9b
> > It is easy to implement, just add a setter method for this parameter to
> MMapDir.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user
configureable to support chunking the index files in smaller parts
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-1741:
----------------------------------
Attachment: LUCENE-1741.patch
Attached is a patch using the JRE_IS_64BIT in Constants. I set the default to 256 MiBytes (128 seems to small for large indexes, if the index is e.g. about 1.5 GiBytes, you would get 6 junks.
I have no test data which size is good, it is just trying out (and depends e.g. on how often you reboot Windows, as Eks said).
> Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
> ---------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1741
> URL: https://issues.apache.org/jira/browse/LUCENE-1741
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1741.patch, LUCENE-1741.patch
>
>
> This is a followup for java-user thred: http://www.lucidimagination.com/search/document/9ba9137bb5d8cb78/oom_with_2_9#9bf3b5b8f3b1fb9b
> It is easy to implement, just add a setter method for this parameter to MMapDir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user
configureable to support chunking the index files in smaller parts
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730547#action_12730547 ]
Uwe Schindler commented on LUCENE-1741:
---------------------------------------
OK, we have two patches, we can think about using one of them.
In my opinion, there is no problem with limiting the chunk size on 32 bit systems. The overhead of choosing the right chunk is neglectible, as it only affects seeking. Normal sequential reads must only check, if the current chunk has enough data and if not, move to the next. The non-chunked stream does this check, too (to throw EOF). With a chunk size of 256 MB, the theoretical maximum number of chunks is 8 (which can be never reached...).
Any other comments?
Eks: What was you value, that fixed your problem without rebooting. And: How big was your biggest index file?
> Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
> ---------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1741
> URL: https://issues.apache.org/jira/browse/LUCENE-1741
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1741.patch, LUCENE-1741.patch
>
>
> This is a followup for java-user thred: http://www.lucidimagination.com/search/document/9ba9137bb5d8cb78/oom_with_2_9#9bf3b5b8f3b1fb9b
> It is easy to implement, just add a setter method for this parameter to MMapDir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user
configureable to support chunking the index files in smaller parts
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730590#action_12730590 ]
Uwe Schindler commented on LUCENE-1741:
---------------------------------------
Javadocs state (in FileChannel#map): "For most operating systems, mapping a file into memory is more expensive than reading or writing a few tens of kilobytes of data via the usual read and write methods. From the standpoint of performance it is generally only worth mapping relatively large files into memory."
So it should be as big as possible. A second problem with too many buffers is, that the MMU/TLB cannot handle too many of them effective.
In my opinion, maybe we could enhance MMapDirectory to work together with FileSwitchDirectory or something like that, to only use mmap for large files and all others handled by NIO/Simple. E.g. mapping the segments.gen file into memory is really wasting resources. So MMapDir would only return the MMapIndexInput, if the underlying file is > X Bytes (e.g. 8 Megabytes per default) and fall back to SimpleFSIndexInput otherwise.
bq. In any event, Uwe can you add to the javadocs describing this false OOM problem and what to do if you hit it?
Will do this tomorrow, will go to bed now.
Here are also some other numbers about this problem: http://groups.google.com/group/jsr203-interest/browse_thread/thread/66f6a5042f2b0c4a/12228bbd57d1956d
> Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
> ---------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1741
> URL: https://issues.apache.org/jira/browse/LUCENE-1741
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1741.patch, LUCENE-1741.patch
>
>
> This is a followup for java-user thred: http://www.lucidimagination.com/search/document/9ba9137bb5d8cb78/oom_with_2_9#9bf3b5b8f3b1fb9b
> It is easy to implement, just add a setter method for this parameter to MMapDir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user
configureable to support chunking the index files in smaller parts
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730373#action_12730373 ]
Uwe Schindler commented on LUCENE-1741:
---------------------------------------
Good idea. Do we have still this 64bit detection property in the utils? If yes, this could be easily done.
> Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
> ---------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1741
> URL: https://issues.apache.org/jira/browse/LUCENE-1741
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1741.patch
>
>
> This is a followup for java-user thred: http://www.lucidimagination.com/search/document/9ba9137bb5d8cb78/oom_with_2_9#9bf3b5b8f3b1fb9b
> It is easy to implement, just add a setter method for this parameter to MMapDir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user
configureable to support chunking the index files in smaller parts
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-1741:
----------------------------------
Attachment: LUCENE-1741.patch
Updated patch with Mike's suggestion.
> Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
> ---------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1741
> URL: https://issues.apache.org/jira/browse/LUCENE-1741
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1741.patch, LUCENE-1741.patch, LUCENE-1741.patch
>
>
> This is a followup for java-user thred: http://www.lucidimagination.com/search/document/9ba9137bb5d8cb78/oom_with_2_9#9bf3b5b8f3b1fb9b
> It is easy to implement, just add a setter method for this parameter to MMapDir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Closed: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user
configureable to support chunking the index files in smaller parts
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler closed LUCENE-1741.
---------------------------------
Resolution: Fixed
Committed revision: 793826
Thanks Eks!
About the automatic fallback to a SimpleFSIndexInput for small files like segment*, *.del, I open another issue targeted to 3.1. MMapping of small files is wasting system resources and may be slower than just reading a few bytes with SimpleFSIndexInput.
> Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
> ---------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1741
> URL: https://issues.apache.org/jira/browse/LUCENE-1741
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1741.patch, LUCENE-1741.patch, LUCENE-1741.patch
>
>
> This is a followup for java-user thred: http://www.lucidimagination.com/search/document/9ba9137bb5d8cb78/oom_with_2_9#9bf3b5b8f3b1fb9b
> It is easy to implement, just add a setter method for this parameter to MMapDir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user
configureable to support chunking the index files in smaller parts
Posted by "Paul Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730551#action_12730551 ]
Paul Smith commented on LUCENE-1741:
------------------------------------
An algorithm is nice if there are no specific settings specified, but in an environment where large indexes may be opened more frequently than the common use cases, then what is happening is that the Memory layer is getting OOM conditions too much, forcing too much GC activity to attempt the operation.
I'd vote for checking if settings have been requested and using them, and if not set rely on a self-tuning algorithm.
In a really long running application, the process address space may become more and more fragmented, and the malloc library may not be able to defragment it, so the auto-tuning is nice, but it may not be great for all peoples needs.
For example, our specific use case (crazy as this may be) is to have many different indexes open at any one time, closing and opening them frequently (the Realtime Search stuff we are following very closely indeed.. :) ). I'm just thinking that our VM (64bit) may find it difficult to find the contiguous non-heap space for the MMap operation after many days/weeks in operation.
Maybe I'm just paranoid. But for operational purposes, it'd be nice to know we could change the setting based on our observations.
thanks!
> Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
> ---------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1741
> URL: https://issues.apache.org/jira/browse/LUCENE-1741
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1741.patch, LUCENE-1741.patch
>
>
> This is a followup for java-user thred: http://www.lucidimagination.com/search/document/9ba9137bb5d8cb78/oom_with_2_9#9bf3b5b8f3b1fb9b
> It is easy to implement, just add a setter method for this parameter to MMapDir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1741) Make MMapDirectory.MAX_BBUF user
configureable to support chunking the index files in smaller parts
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-1741:
----------------------------------
Fix Version/s: (was: 3.1)
2.9
> Make MMapDirectory.MAX_BBUF user configureable to support chunking the index files in smaller parts
> ---------------------------------------------------------------------------------------------------
>
> Key: LUCENE-1741
> URL: https://issues.apache.org/jira/browse/LUCENE-1741
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1741.patch
>
>
> This is a followup for java-user thred: http://www.lucidimagination.com/search/document/9ba9137bb5d8cb78/oom_with_2_9#9bf3b5b8f3b1fb9b
> It is easy to implement, just add a setter method for this parameter to MMapDir.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org